Results tagged “Corporate”

Until today morning I thought that the upcoming two weeks will be easy ones because my teammates are on holidays and without them I'm almost unable to do anything related to our customers or something with connections to databases.

But when I sat down on my workplace stress level almost immediately raised sky-high because we had a system failure at a customers site.

Exactly right to hit their peak load on monday morning, great!

After some minutes we saw that the system was slowly collapsing because errors were accumulating and building up each other.

Well, in the end we managed it by switching off half of the customers network and letting errors settle down.

Later I discovered that at the beginning of the whole mess was a not-responding database causing timeout-errors and so on. But our database experts still have no clue why the DB was not responding. Weird.

Hope, this doesn't happen again...

Nevertheless, we could have managed much better if the customer had upgraded to a newer version of our software which has enhanced failover algorithms and error supression.

|

Today I had the meeting with my department leader.

Initially it was planned for one hour but after a bit more than two hours we decided to continue on friday.

The chat was quite informative for me and also for him because I now know much more about what's going on in the planning and management levels and he knows more about how the situation and impressions on the development levels are.

All in all most many of my questions and concerns are worked on somewhere in the management but below them only few know about that.
He now knows, that this is an issue here.

It was noted that some more information about what's going on with us should be communicated down to us developers. In my opinion this would help much to lift the mood and I wouldn't have asked quite some of my questions.

Some more on that on Friday, currently I'm quite satisfied with the answers I got and I think we both are benefiting from the talk.

|

Only hours before I went on vacation two weeks ago, my HDD on the main development machine failed. I started a backup then which was finished by my group leader on monday.

Today my replacement HDD began making noises too, a quick S.M.A.R.T. check showed me, that this one was failing too.

And now I'm backupping again, second time in two weeks. Something is seriously wrong with my working equipment...

|

So far I have not received any response on my mail. The talk today was delayed until next week when both responsible people are present and have time for me.

On the meeting today my mail was shortly mentioned on a point that the corporate communication has to be improved because the CEO has "received a mail with many questions, good questions, where many have a quite simple answer and shouldn't have to be asked when communication would be right".

And of course, some of my questions got a partial answer during the meeting but there are enough left.

I also talked to more of my colleagues and showed them my mail and all said that it's written a bit "aggressive" but nevertheless hits the reality.

|

Yep. I'm moving something.
I sent out my questions for the All Hands Meeting two hours ago and I just got a call from my team leader that the management level of our company is currently quite busy and thinking about what to do with my mail.

My department leader who is on vacation too got calls from several positions and now he is afraid that my mail can affect the whole development department, I'm told.

Well, on Wednesday when I'm back at work, just right for the meeting on that afternoon, he wants to talk to me.

Update:
I got another call from my group leader and then from the department leader. Seems, I'm really causing some confusion in the upper levels of the company. And I'm causing much stress for my department leader. He isn't really angry on me but he said I shouldn't write such mails again without consulting him before. Tried that today, but he didn't pick up the phone... Next time I'll try to get such mails ready earlier.

And I'll invite him on a beer as soon as he's back from his vacations.

|

I'm on vacation next week, don't know if and how often I'll update here.

On the 22nd or 27th there will again be a company wide meeting, like the last Town Hall Meeting, but this time it's called "All Hands Meeting".
I'm currently setting up an email, which I will send the companies heads a few days before the meeting, describing the actual situation for developers and asking a few questions.

Perhaps I'm getting uncomfortable for some people but that doesn't bother me. What I want to reach are better conditions for all developers, better product engineering environment and a higher motivation for everyone.

Keyword "motivation": It has been quite low for some weeks now and it doesn't help that again a few people left our location last week theirselves, one being quite a (frustrating) key-position in one of our top-projects, the other one wanting to try something new.

But what I also expect is, that our company will be unable to change conditions in the short term because we have a lack of qualified engineers resulting in developers burried into work.

Anyone out there searchig a job? Work conditions are generally quite good if you don't care on some questionable managements decisions and are resistant against rising stress-level near the end of most projects (which I think is the case in any company out there anyway).

|

Would anybody on earth be so "clever" and sell a product to a large-scale provider with "unlimited licenses and unlimited machines"?

Well, if you are in contact with one of our sales(?)-men, you can get something like that.

Some days ago a contract went out to a customer containing exactly this clause.

So, if this customer ever wanted to spread all over the world and gather millions of customers with our product, we would still have to be happy with the ~70.000 bucks we earned with that contract. But we would have to maintain all the systems the customer needs to be able to host millions of customers. Clearly a no-win situation for us, regardless of which perspective you look at it.


Dammit! And such people are in the UPPER levels of our company! Why do sales people always have to think only how to get a signature below the contracts and never think of the future?

|

Yesterday evening I got a call from our maintenance team on the attendance phone (I'm responsible this week).

A customer had problems with our product.

After some initial investigation (all by phone, because I wasn't at home) it was clear that it was a problem with the database (of course *sigh*).

We fixed the problem within 15 minutes. To my own surprise I knew enough of the database interna to tell him what cause lead to which problems and where to look for further informations.

It was somehow the customers own fault. He has some problems in his network which we have told him for months now, but the customer had no time (or didn't care which I think is a better cause) to fix it.

Well, now it has been communicated over official lines and I hope, they fix their problems faster now.

|

Fastest release ever, 38 minutes including testing. Yeah.

Well, it just went so fast because shutting down the old software took only seconds instead of up to half an hour because it crashed during the shutdown, but I don't think that anyone noticed much. Downtime was less than 40 seconds.

|

Yeah, today (in 5 minutes) there is again a release at a customer.

But this time it'll just a small one, no dangerous DB changes involved.

Should be finished in about 1.5 - 2h.

|

Over the weekend there has been a power outage at the companys location.

I recognized that this morning when I had to re-logon on my two Windows-stations and my Linux-station was completely down.

While the Linux and one Windows computer got up again without problems my working Windows computer spit out lots of filesystem problems and unknown errors (during chkdisk???).
Of course it was my main Windows computer.

Surprisingly I got a new HDD for backup almost without problems and within 30 minutes. Now I'm installing Win2k again on it and then I'll back up everything what's left from the old HDD before secure erasing and giving it back.

Again, almost a whole day just setting up computer because of power fluctuances.

|

No specific topic currently, just a few points. I now get paid out a lot of my overtime hours and keep record of the new ones quite strictly. It seems that my department leader is a bit surprised now how many people do overtime hours for the various reasons.

It's strange, you should avoid overtime hours, but get your work done for specific dates.

Yesterday the machine cluster where TaskZilla (and other of our internal tools, as i.e. TWiki) sits on suddenly disappeared. Later I found out that one of the Sysadmins wanted to do something with the UPS (Uninterruptible Power Supply) of the cluster and suddenly the cluster stopped making noise ;) There was some trouble during re-startup of TaskZilla like missing mountpoints, wrong database-links, wrong webserver config, etc., but after about 1 1/2 hours, everything was up again. Luckily the database files seem to be unharmed.

Just a simple switch has died in the electronic shock, so TaskZilla is not reachable from some networks anymore, should be fixed by noon today.

Today will be a meeting with some project managers and development managers. The meeting has been called in because a customer wants an upgrade of its current system to support a new technology. The problem is that we can't (well, we can, but it's too risky in everyones eye) replace the whole system with the new one but we have to integrate the new one into the old one. Of course there are many small things to do and we already guessed their duration but aparently the overall duration is too long for the project managers and today in the meeting there will be a discussion how to lower the development time or invent another solution which takes less time.

But I don't think that the time can be lowered that much. There is work which has to be done and we already told how long it would take. We can't do it faster just because the cost is too high for the project manager. Either we take the time for that or it won't be done in an acceptable way. In the first case the cost is too high in the last the customer will have much trouble and we will need more time again to fix these troubles.

Of course I personally want to make it right, because on the long run it gives the customer a better quality software with less trouble but somewhat higher cost. And then he hasn't to pay afterwards for upcoming bugfixes and repairs.

Dammit, there is no clear solution which would satisfy the project manager AND the developers.

And (of course) the initial development should be finished in a month and without overtime hours ;)

I personally want to hold the deadline in a month to give the customer a great experience but that would need an almost uninterrupted dedication from our side which is more than improbable. And it would be too expensive for the project manager ;)

|

Whoa, the update yesterday took much longer than expected.

I went home at ~09:00. Upgrading the database caused much trouble and took more than twice as long as initially planned. It was not delayed because we were not prepared well enough but because we stumbled across an Oracle internal bug which caused new columns in tables to be inaccessible the first time. You have to remove it and recreate it again. This had to be done by hand and couldn't be accomplished by our prepared scripts. But the DB guys managed quite well.

On my side (OS, application and shellscript update) I had some trouble because every machine which had to be updated was different from any other. Different existing packages, different running daemons, different libraries... Hell, they should all be the same!

Then we had some more problems with a misconfigured hardware piece and had to call in additional assistance.
After that some important passwords were invalid, etc.

All in all we just missed the expected end by just about 5 hours ;)

But I want to thank my team which mastered all upcoming problems regardless of their complexity. Great work, this had been the largest update of our platform ever, and we are operational with just a few hours shift.

|

Well, in a few minutes another release of our software takes place.

This time it is a bit more complicated than other releases before because we're upgrading a more than two years old system to the up to date version.

This involves large parts of database migrations, web interface configurations and OS updates.

Should take ~2 - 4 hours.

Let's get it on!

|

Two days ago I managed another release to a customer.

Initially everything was well prepared and planned, all people were confident that everything will go well and so we started the update at ~19:00.

Just a few minutes later I had to correct my timeline the first time. A interface on the target machines had changed since the afternoon and I had to quickly hack in a fix and throw out a new bugfix.

So good so far, applications released, now let's test the bugfixes... Click here, change that, ... HU? Where's the data?
Data disappearing, and after some time fiddling, we had to call in an expert for our WebInterface because we expected something wrong with it.

And in the meantime we found some database inconsistencies, great.
I had given up correcting my timeline.

When our web expert arrived he quite quickly found out that the files, he released on the afternoon on the machines, had changed and downgraded to a version of last november?!?
So update them again and now it's working as expected.

In the whole, the release lasted almost three hours, double as long as planned, but the customer is completely satisfied.
My team managed to solve all problems and worked around all difficulties and in the end we got the expected result.

Later it turned out that another team had a release of another product to the same customer and accidentially overwrote some of our files.
No team knew of the others release, someone forgot to tell us so that we could have synched our work.
Shouldn't happen again, people are now more aware of how our different teams can interfere.

|

Once again I'm sitting here at my workdesk and we're completing the release last friday. "Completing" because we only installed the new version on one of the two machines to give it a try during the week and pull the other to the actual version if it proves to work well.

It worked well, we're almost done now.

Last release duration was about 1 1/2 hours, this one will be even shorter. I'll expect a "Everything OK" in a few minutes, and we began update not an full hour ago yet... Yeah, go baby!

|

Whoops, seems I've been raising dust today.

The situation is, that I've gathered ~250 hours overtime meanwhile and I thought that it would be nice if I got paid them out partly.
I chatted about that with my team leader and what possibilities I could have if I asked our department leader about that.

What complicates the stuff is that I found a paragraph in my collective contract which states that I can press on all hours over 154 to be paid.

My team leader then chatted with my department leader along the way and aparently there is some movement in the company to get somehow rid of the overtime hours, but long time comp time is not the ultimate solution...

|

After quite a while of debugging and testing (we got permission to turn on a bit more logging on our customers machine) I found out that the disappearing data is received by the second component but not processed.

Yesterday evening I found out why:
The processing of the data in our classes is done with the help of a Visitor Design Pattern. This was engineered by a former collegue of me but not in a easy recognizable way.

And so it came that a very specific situation could occour and the data slipped through all processing untouched.

Fixed it and after a new release (presumably on next monday evening/night), we should be down to 0,000005 percent of missing data. 10 records in two million which still disappear without any apparent reason.

And tomorrow night we'll fully convert the system where we partially released last friday.

|

This had been a strange weekend.
The release on Friday went without any notable problems and everything worked quite well.

But on Saturday evening my group leader called me and I had no good feelings. But the new release still worked and he called because of another problem at another customer.

It seems that we're loosing data between two components of our product but there is no sign of an error or failure in any of the logs, which is theoretically impossible.

But it is possible to reconstruct the missing data and that's what we did from 10:00 till 16:00 yesterday on Mother's Day.
It's just a huge time-consuming, yet semi-automatic, task.

Dammit, why are 0,0001 percent of the data hiding? Does someone delete them by hand, just to give us trouble?

|

And in half an hour our third try to update one of our customers systems happens.

Hope, this time there are no more errors...

|

  1 2 3 4 5 6 7  

Archives