Thursday, February 03, 2005

The sound of dying disks...

I've pretty much decided I should have stayed in bed this week...

At the start of the week I discovered that after the bad disk in our main server got replaced around the middle of last month, the backup of our CVS archive got restored into two different places due to a absence of a symbolic link. For historical reasons, different eSTAR team members have been using different CVSROOT paths, and with this symbolic link present everything was pushed to the real CVS archive without people noticing. However, with two "real" copies of the archive rather than one copy and a symbolic link pointing to it, half the team started writing to one version of the archive, while the other half wrote to an entirely different version of the archive. The result was two "master" CVS archives. This was not a good situation, although fortunately the resulting merge conflicts when we synchronised the two archives turned out to be fairly minimal.

Today, the primary disk on our development server died. I've spent most of the day, along with our system administrator, putting the server back together again from backups. It's not been a good day, or even a good week. At least we had good backups, that's something to be thankful for, it could have been a lot worse.

Of course, now I'm waiting for the third shoe to drop...