Easy weeks?

Until today morning I thought that the upcoming two weeks will be easy ones because my teammates are on holidays and without them I'm almost unable to do anything related to our customers or something with connections to databases.

But when I sat down on my workplace stress level almost immediately raised sky-high because we had a system failure at a customers site.

Exactly right to hit their peak load on monday morning, great!

After some minutes we saw that the system was slowly collapsing because errors were accumulating and building up each other.

Well, in the end we managed it by switching off half of the customers network and letting errors settle down.

Later I discovered that at the beginning of the whole mess was a not-responding database causing timeout-errors and so on. But our database experts still have no clue why the DB was not responding. Weird.

Hope, this doesn't happen again...

Nevertheless, we could have managed much better if the customer had upgraded to a newer version of our software which has enhanced failover algorithms and error supression.


Similar entries

These entries are similar to this posting (as of 2017-02-06):