20141225

Behind The Firewall: Connecting Service Despite Restrictions

Here's the problem:

You have spent a lot of money for some internet service, and being the good administrator you are you'd like to share this service among your very small collection of machines.  You also want to put a firewall in the way of unnecessary and/or unwanted network traffic, because after all, the more hoops people have to jump through to hit your systems, the better.

Now, you've got your firewall connected, and you can ping the world and even browse to places on the firewall.  But when you connect a system behind the firewall, every request out seems to die.  You've checked that the firewall was properly configured, and you had even tested it back at the office.  You can still reach the world from the firewall, and yet from your client machine you can't go anywhere.

After a Wireshark capture of the traffic, you see your client machine traffic going out, and some interesting ICMP messages that say "Time to live exceeded in transit" or "TTL expired in transit."  What's more, you notice that your firewall is sending these messages back out to the servers you're trying to talk to!  It is as though no matter what comes in, its TTL exceeded.

And that is exactly what is happening.  Let's create this situation on purpose.  If we have a NATing firewall that is to be the only NAT device in the network, we don't want to allow any other NAT devices to deliver their traffic to their clients.  It's endpoints or nothing at all.  In iptables, the following rule should accomplish this:

iptables -t mangle -A PREROUTING -i eth0 -j TTL --ttl-set 1

(Refer to this page for more information: http://www.linuxtopia.org/Linux_Firewall_iptables/x4799.html)

The above command should mangle the TTL such that there is only one more hop it can go before it runs out of life.  I haven't tested that - if it doesn't work, try 2 instead of 1.  The above command is really based upon the fix for the problem the above command causes.  Basically, if you're experiencing an issue with the TTL expiring too soon, just rewrite it!

iptables -t mangle -A PREROUTING -i eth0 -j TTL --ttl-set 255

Now we can do another 255 hops before we run out of TTL.  As this is done in the prerouting, the packet is fixed before any checks are made as to whether or not it should be dropped for TTL expiration.  A similar fix can be done with pfSense (search in google for the appropriate terms and make sure to tread carefully when mucking around with the filter code - note that there IS NO GUI OPTION FOR CHANGING THE TTL IN PFSENSE!!).  An alternative to the above is to increment the TTL by 1 or more.

If you have a pfSense VM running on a Linux hypervisor, you can make the fix right in Linux (for the bridge adapter, of course).  IPTables in this way really saves the day.  The fix in pfSense is not terrible, but also not convenient and is not officially supported by their community.

Now that I think about it, this would be a great way to stop people from using unauthorized wireless access points around the office....

20141216

Never run updates from recovery...

Concerning Ubuntu Server 14.04 (and possibly others)....

It's probably not something you should do.  I don't know what Ubuntu's official stance would be, but my experience has been that it's not the best thing to run apt-get dist-upgrade from a recovery kernel.  There are a few things missing, even if you mount the root partition:

  • /dev, /proc, /sys and the gang are all there, but only sort of.  And /var/run?  Well, not really this go-around, if you're using 14.04 LTS.
  • Upstart didn't run from this root, so there is no Upstart.  Anything that needs to deal with Upstart (like, evidently, libpam-systemd) will fail.
  • If something is broken, it might get more broken if you run updates in recovery.

Still, it may become necessary.  There might even be some failures that look semi-legit but are certainly not going to help you back to a working system, because said failures are standing in the way of completing your updates.  One such issue is when one of the update scripts for one of the libraries being updated decides - suddenly - that it really needs to run a script out of /etc/init.d/.  Let's suppose, whatever the reason, no such script exists at the time of this update.  Apt fails, reporting that one of its packages wasn't able to complete all its steps.

One possible solution to this is to trick Apt into thinking there really IS a script available.  In my case, I simply:
  1. touch /etc/init.d/systemd-logind
  2. chmod +x /etc/init.d/systemd-logind
Viola!  Apt was able to finish processing the updates and now I might have a working system again.  I stress might because it shouldn't have broken in the first place and I have no earthly idea why it did break.  But, obviously, missing or non-configured update packages might be in the recipe.

The important thing, when you perform something abominable like what I mentioned above, is that you remove it before you reboot.  If the service does depend on Upstart, having the fake init.d script available may cause much pain.

And that's that.