One of my servers, foggy, is routed to the outside world via a DSL line instead of my 2mb leased line. This is so that I can do backups of my servers in Cheltenham, to here, without saturating my leased line. And, of course, the DSL is three times faster.
So I woke up today to a voice saying "One hundred and sixty eight alerts". I looked, and they were nearly all foggy, complaining of being unable to contact the world outside. So I looked at it. Foggy couldn't ping the DSL router. So, first thing, I told the router to reboot. That didn't fix it. So I used my remote power thingy (an APC PDU) to power-cycle the router, That didn't help. Foggy couldn't ping the router, but I could ping it from another server. So, I thought, that probably means a problem with foggy, rather than a problem with the router.
So I restarted foggy's network with "/etc/init.d/network restart" then re-established the gateway to the DSL router with:
/sbin/route add -net 0/0 gw 10.149.17.254 em1
/sbin/route del -net 0/0 gw 10.149.17.1 em1
and everything worked. I have no idea what happened here - foggy just seemed to have forgotten where the DSL router was?