It started at 9pm last night. Suddenly, my 2 mb leased line was dead, and all my computers were reporting that they couldn't access the internet. Bad. Very bad. So I called Daisy Communications.
"How long has it been down," they asked. About two minutes, I explained - my monitoring system actually works; theirs doesn't. They checked and found that they couldn't ping my router, so they passed the problem on to Vodafone, who are the people who actually run the network. After a few hours, I decided that this was going to be a long one, so I went to bed.
At 9am the next morning, it was still down. Daisy confirmed that Vodafone were working on it. It was one of their core routers.
That's like Piccadilly Circus and Trafalgar Square. If those are closed down, London becomes one huge traffic jam - nothing can get through. That was good news, though, because it meant that they didn't have just one whiney customer bleating at them, they had a MAJOR problem, and hopefully would throw major resources at it. Wouldn't you think?
So today it was Sunday, and we took the train to London to visit daughter.1 and grandson.1. About ten minutes after I got on the train, I got a call from Daisy. Vodafone have fixed the problem, but Daisy can't contact my router, could you reboot it, "No, I'm on a train." "Oh." "And anyway, I could reboot the router, but not the BT equipment, because that gets power from the phone line, and there's no off switch." "Oh." "And anyway, there's no way the problem could be the router, because it isn't connected to Vodafone''s equipment, it's connected to the BT Box." "Oh. Well, could you reboot the router?" Sigh. "OK."
So here I am on the 11:12 from Chalfont and Latimer to Marylebone, and I want to power cycle my router in Little Chalfont, and I don't have a remote reboot facility for that, because if I need to reboot the router to get onto my network, the problem is that I can't get onto the network to tickle a remote rebooter. But there is one chance. It's a million to one, but it just might work ...
Ruth comes in on Sunday to help with the cleaning. So I phoned home, and she answered. I explained what I wanted her to do, and explained that as nothing was working, there was nothing she could do that would make it worse. We used her mobile; I talked her into the room where the router was, and described the table it was on. Actually, on that table is the firewall and the router, and I couldn't remember which was on top, but that wouldn't matter. She looked at the front of both of then, and saw green lights. I told her to look round the back where the power switches were, and she switched them both off. She checked that the power lights were off. Then I told her to switch them back on, she did, and check that the power lights were on. Thank you Ruth, you are now a sysadmin.
This sounds easy, but remember I'm on a moving train, I'm working from memory to do something I never expected I'd need to do with someone whose job description does not include "sysadmin", and there are tunnels on the line during which there's no telephony.
So then I phoned Daisy to tell them that the power cycle was done, and they confirmed that there was still no contact. Which meant that, as I'd told them, rebooting the router was futile. So they got back to Vodafone and told them that the reboot didn't do the trick, and they need to take further action.
Remember, this is one of their core routers. A lot of people are without internet access while this is down.
I went to lunch with daughter.1 etc, we had a Mother's Day lunch at Pizza Express, and when I got back to her flat, I fired up the Dell laptop that I keep there for just such an occasion.
It wouldn't start, The CMOS battery needs to be replaced, so it had lost its setup info. So I told it the date, guessed a time and told it to reboot, and it did as it was told. Then I tried to contact my network, by trying to access my Secure Server remotely, because that, for obvious reasons, is accessible to the whole world, so they can give the needful info securely, It wouldn't connect. Rats. Nor could I ping it. Double rats.
I was just about the call Daisy and give them an earful, when I thought, hang on, I can't ping it because I've blocked ping to my secure server, as per the PCI DSS recommendation. So I pinged another of my servers, and it worked! Which means that Vodafone have fixed the problem that was stopping the leased line from working, hurrah.
But why can't I access my secure server?
The next thing was to log into the secure server to find out what the problem might be, but I can't, because people outside my network can't log in to the secure server, because security. But I can log in to it from inside the network. Except that right now, I'm not inside.
So I need to log in to a server that's inside my network (and from there I could log in to my secure server). But my firewall doesn't allow people from outside to log in to the inside, because security
Usually, when I go away for a few days holiday, I relax that a bit, so that there's one computer that can be logged in to from the outside, and then from there I can log in to any other computer on the inside. But normally that facility isn't there, because security.
I won't tell you what I did next, because I don't want anyone else to do it. It was pretty clever, I thought, and it let me log in to one of my Raspberry Pies, which is inside the network, and from there I could log in to any of the other computers. I'm in!
So I went to my secure server and checked. A) Apache was running, and B) I could access it from another computer via the https protocol. Which meant that it was actually working, and I could see that it had collected some user data as of 16:30. So why couldn't I access it?
After some thought, I realised why, This Dell is running Fedora linux core 9, which is a five year old version. That's because I tried to install a more recent version and it wouldn't install.
When a browser contacts a secure server, they negotiate an encryption protocol, and between them, they choose the most powerful encryption that they can both handle. Because of vulnerabilities found recently, I had told Apache not to use SSLv1, SSLv2 or TLSv1. But this old version of Firefox doesn't have the most up-to-date protocols, so it wasn't able to talk to the secure server. When I told the server "Yes, you can use TLSv1" everything worked.
And now I'm a happy bunny.
Computer problems are rarely simple.