I knew that I had a problem when my bandwidth monitor started to tell me that about three times as much data was flowing along my line as was possible. Obviously, the monitor had to be wrong.
I very quickly tracked the problem down - the system drive on the main server was failing. Lots of read and write errors. And that's very annoying, it was a new install; I opened the plastic wrapper on the drive a couple of weeks ago.
And then things got worse.
I tried checking the cables, I tried rebooting, nothing helped. Clearly I had to replace the drive.
First, I switched the load onto a backup server. That's very easy; I just change a couple of lines on my firewall, and all accesses are directed to the backup server.
Then I tried to replace the drive.
My first idea was to use a 2.5 inch Sata SSD, because the server (a Dell Poweredge R805) has a couple of slots at the front for 2.5 inch Sata drives. But the server wouldn't acknowledge that it was there, and when I opened up the server, it was obvious why. The slots for the drives were there, but there was nothing connecting them to the mainboard. I'd need an interface card, and it would have to be a Dell branded card, and the cost would be astrological.
So my next thought was, replace the drive with another new drive. I then spent an hour on that. There's only one Sata connector on the mainboard, so I used that for the DVD drive I use for installing Linux. The drive to install on, would be connected to an interface card that lets me put Sata drives on a PCI-E interface. But that didn't work, because the Linux installer refused to recognise the drives.
And then things got really tricky, because I had to leave to go to a family event. My aunt Kit died a few days ago, at the age of 99, and she was one of my favourite aunts, so we went to the funeral and then back to her daughter (my cousin) for a major nosh-up. So for the next six hours, my backup server carried the load (and hardly anyone noticed).
When I got back, I had a plan. First, I connected the DVD drive to a USB port. Then I removed the PCI-E cards, so all I had was that DVD drive, and the drive I wanted to install Linux on, connected to the motherboard Sata port.
That worked! And several minutes later, I had Fedora Linux version 24, 64-bit on the hard drive. So I replaced the PCI-E cards, connected up the other drives, rebooted and everything was fine. And I have a list of things to do to configure the server the way I want it, and a copy of all the files that I needed to do the configuration.
So about an hour later (plus a couple of hours messing about fruitlessly before the family event, plus six hours at the family event) the server is up and running nicely.