One of my servers at Cheltenham was acting the goat - lots of errors being reported on drive sda, which is, of course, the boot drive. So I decided to apply the well-known power-off-power-on cure. It works often enough to be worth trying.
So I went to my remote power controller ... and couldn't access it. Ugh. So I phoned up Cheltenham, got an immediate answer, and was told, "Oh yes, we changed all the IP addresses for the power controllers." "Hey!" I replied, "How come you forgot to tell me?" Duh.
So they power cycled the server for me, and told me the new address of the power controller, and when I logged into it, I noticed that all the servers were powered up, which A) isn't how I leave it, and B) costs someone electricity, and that's a real cost, I pay £17 per amp per month. They swore that changing the IP address couldn't have caused that, and they think what happened was, some helpful sysadmin noticed that most of the servers were powered off, and helpfully powered them on.
Oh well. Not a big deal, especially as they've agreed not to charge me for the power (and anyhow, we're tallking quite a small amount of money). But it's the principle of the thing, I really don't like third parties messing with my servers without my say-so. So, I was up till 2am on Saturday sorting all this out.
Back to the server that was acting umpty - the drive really is hopping on one leg, as we say in the trade, and will need to be replaced. The other five drives on that box seem OK, so I'll build a server with a new drive, take it down there, and plug the other five into the new box.
While messing with the other servers, I found another one has completely given up the ghost, and I already had four that were needing to be replaced, so it's worth making a trip down there for a big swap-out; six out and six in.
And then, Sunday morning ... one of my servers here started misbehaving. Also traced to a dead drive. So I swapped that out, and restarted the machine ... at which point, another dead drive revealed itself. So another swap-out, and now it looks OK, except that I have to reload those two new drives from backups.
Come Monday, I'll be calling up Bluepoint and giving them the good news. I need 11 new 2-terabyte drives, that'll be about £1000. And a new DVD drive, because the one in my Windows XP tower is on the blink. Yes, I do run Windows XP - I have two geocaching applications that need Windows.