First, a little lesson on how networking works.
When a computer needs to communicate with another, it shouts down the ethernet cable, saying who it is and who it wants to talk to. That works fine for the computers on the same local network. In my case, the local network is all computers with the address 10.x.y.z, because all my computers are on that 10 network, because that's a bunch of addresses that anyone can use, but it's only visible from inside my network; anything outside my network can't talk to them. Which is how I like it!
But what if a computer wants to talk to somethng outside of the local network? For example, suppose it wants to talk to 220.127.116.11 (which is google). Then it doesn't just shout down the network to 18.104.22.168, it needs to talk to the gateway. In my case, the gateway is my firewall, which is on the address 10.0.0.1 (so that all my 10.something computers can talk to it). And then the gateway will forward it to the outside world (via my router). So when I do a route -n command to list the routes that the computer knows about, I get this:
Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 em1
0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 em1
What this means is, for any computer with the IP address 10.x.y.z, that's on the local network, so just shout. For anything else, the gateway is 10.0.0.1, my firewall.
OK, that works fine. But today, suddenly, it lost that last line, which meant that anything outside my local network, was suddenly unavailable, because it didn't know where the gateway was - to be more precise, it had forgotten that there was a gateway. And it also did the same thing, suddenly, last week.
The cure is simple.
route add -net 0/0 gw 10.0.0.1 em1
That adds the route to the gateway back.
This isn't very satisfactory, because it means that each time it forgets about the gateway, I have to tell it again. I've set up a cron job that tests, once per minute, to see if 22.214.171.124 is reachable, and if it isn't, do the "route add".
What I don't understand, is how come this server (and it is only this one) forgets about the gateway?