Balancing on a knife's edge
Today I accidentially did something which easily could have cut my head in company.
Tomorrow we have a release at a customers place, where we bring up an additional machine to handle the increased load.
I was creating configurations for the new machine in the administration interface when it failed and just returned an error. I was forced to directly connect to the database to create the entries. During the creation I stumbled over some small errors and decided to throw away the whole table data and copy everything from the master database when I realized that I actually was working ON THE MASTER DATABASE! I had thrown away the master data, dammit. Adrenaline... Sweat... Has someone seen that? No? Good. Ok, hopefully none of the running machines restarts to fetch its now vanished config. What do I do now? Re-fetch the data from one of the client databases. Dammit, replication has already deleted the data there too.
More sweat, more adrenaline...
But wait, what was one of those minor errors above? Whoa, replication to one machine was failing, the data could still be there... Immediately connect to that machine, stop the replication mechanism, check for the data. YEAH! Still here, whew.
Copying back the data from that failing machine was easy and luckily no one noticed that incident.
I have to be much more careful in the future, when I deal with databases. Have to get practice on that...