Server down!

I woke up this morning and saw my scrolling display telling me that I had a big server problem. Sure enough, one of my bigger servers at Cheltenham was down.

I tried rebooting it, and got a bit of a response, but after a little while, I decided that it was no go. But worse - this was the backup server of the server that went down a week ago. I was still loading up the two replacements that I was preparing, and they weren't ready yet - I was expecting them to be fully ready some time next week or so. Nevertheless, I decided that I needed a trip to Cheltenham today.

So I loaded up the two new servers; they've been running for a week without problems, so although they weren't fully loaded yet, at least they'd been well tested. And some tools, and some cables. And a spare hard drive for the server that had just gone down, and a spare power supply for the one that went down last week.

The trip to Cheltenham was easy, it being the weekend, and parking was great. We got the dead servers out of the rack, and I opened them up. As I expected, the one that died last week was a power supply problem; as soon as I replaced that, it was good to go. And the other one was indeed a hard drive issue, so I replaced that.

So now instead of two big servers (one and a backup) I have four. And instead of each of them having four 2th drives and two 3tb, they each have six 4tb drives.

So things are back in action, and I can continue to load up the two new servers. Time spent in Cheltenham - two hours, but add another four hours for driving there and back. And that's why I try to organise things so that I rarely need to visit.