Port Forwarding FTL
OK, so we had a bit of a situation. We have a full rack of co-located production equipment, running a critical web application which can’t go down. The trouble is, the router and firewall belong to someone else and it’s getting turned off at the end of the week.
Not a problem, you cry, just get your own connectivity and firewall/router and hook em up. That’s fine. Until you get to the point where you have to change DNS to put it live.
So here’s the layout. Old production firewall is box A. Whole production system is a big virtual box B (with loads of machines in it). New production firewall, on a different external address, is box C.
Currently, traffic comes into box A, port forwards to box B, which has box A as its default route. All fine and dandy. Trouble is, we want to change DNS so that traffic is routed to box C, which port forwards to box B, which….. sends its response out through its default route on box A, never to be seen again.
“You thick muppet,” you cry, “why not just change the default route on box B?” Well, remember that bit about no downtime? Just as connections to box C don’t work with the default route set as A, connections to A won’t work with the default route set to C – making things a little tricky during the DNS broadcast when the system effectively needs to answer to two addresses at once.
There is, naturally, a solution – a filthy FILTHY hack which makes the firewall look like the plate of noodles above. The core problem is this – traffic sent to an internal machine in box B leaves by the wrong route. Traffic on box C, however, leaves by the correct route. So, the plan is to make the traffic on box B “happen” on box C instead.
What?
Ok, here’s how it’s done. SSH TUNNELS! LOTS OF THEM! ONE FOR EACH SERVICE!
Look, if I forward port 80 to the internal machine, the traffic gets lost, right?
But the internal network is fine because it’s a flat class C – it’s only connections which require the default route which have issues. So, we do the port forward in two stages. First of all, we set up an SSH tunnel so that port 80 on the web server is forwarded to, say, port 12345 on the firewall’s internal interface. Then we follow up by forwarding port 80 on the firewall to port 12345 on the internal interface of the firewall, which transparently tunnels it to port 80 on the internal web server. Hooray!
Of course, this is a hack on a kludge of almost biblical proportions, but it only needs to be in place until the DNS has propagated, at which point we can change the default routes to point to box C, drop the tunnels and do the port forwarding properly.
So, as such, it’s not REALLY a filthy hack – it’s a piece of creative networking designed to avoid downtime while re-homing an entire system.
I want a job in academia so I can impose periods of arbitrary downtime and save having to turn my networking configs into chop suey.
Leave a Comment