Back @ work
Whoa, the update yesterday took much longer than expected.
I went home at ~09:00. Upgrading the database caused much trouble and took more than twice as long as initially planned. It was not delayed because we were not prepared well enough but because we stumbled across an Oracle internal bug which caused new columns in tables to be inaccessible the first time. You have to remove it and recreate it again. This had to be done by hand and couldn't be accomplished by our prepared scripts. But the DB guys managed quite well.
On my side (OS, application and shellscript update) I had some trouble because every machine which had to be updated was different from any other. Different existing packages, different running daemons, different libraries... Hell, they should all be the same!
Then we had some more problems with a misconfigured hardware piece and had to call in additional assistance.
After that some important passwords were invalid, etc.
All in all we just missed the expected end by just about 5 hours ;)
But I want to thank my team which mastered all upcoming problems regardless of their complexity. Great work, this had been the largest update of our platform ever, and we are operational with just a few hours shift.