Behind The Firewall: Connecting Service Despite Restrictions

Here's the problem:

You have spent a lot of money for some internet service, and being the good administrator you are you'd like to share this service among your very small collection of machines.  You also want to put a firewall in the way of unnecessary and/or unwanted network traffic, because after all, the more hoops people have to jump through to hit your systems, the better.

Now, you've got your firewall connected, and you can ping the world and even browse to places on the firewall.  But when you connect a system behind the firewall, every request out seems to die.  You've checked that the firewall was properly configured, and you had even tested it back at the office.  You can still reach the world from the firewall, and yet from your client machine you can't go anywhere.

After a Wireshark capture of the traffic, you see your client machine traffic going out, and some interesting ICMP messages that say "Time to live exceeded in transit" or "TTL expired in transit."  What's more, you notice that your firewall is sending these messages back out to the servers you're trying to talk to!  It is as though no matter what comes in, its TTL exceeded.

And that is exactly what is happening.  Let's create this situation on purpose.  If we have a NATing firewall that is to be the only NAT device in the network, we don't want to allow any other NAT devices to deliver their traffic to their clients.  It's endpoints or nothing at all.  In iptables, the following rule should accomplish this:

iptables -t mangle -A PREROUTING -i eth0 -j TTL --ttl-set 1

(Refer to this page for more information: http://www.linuxtopia.org/Linux_Firewall_iptables/x4799.html)

The above command should mangle the TTL such that there is only one more hop it can go before it runs out of life.  I haven't tested that - if it doesn't work, try 2 instead of 1.  The above command is really based upon the fix for the problem the above command causes.  Basically, if you're experiencing an issue with the TTL expiring too soon, just rewrite it!

iptables -t mangle -A PREROUTING -i eth0 -j TTL --ttl-set 255

Now we can do another 255 hops before we run out of TTL.  As this is done in the prerouting, the packet is fixed before any checks are made as to whether or not it should be dropped for TTL expiration.  A similar fix can be done with pfSense (search in google for the appropriate terms and make sure to tread carefully when mucking around with the filter code - note that there IS NO GUI OPTION FOR CHANGING THE TTL IN PFSENSE!!).  An alternative to the above is to increment the TTL by 1 or more.

If you have a pfSense VM running on a Linux hypervisor, you can make the fix right in Linux (for the bridge adapter, of course).  IPTables in this way really saves the day.  The fix in pfSense is not terrible, but also not convenient and is not officially supported by their community.

Now that I think about it, this would be a great way to stop people from using unauthorized wireless access points around the office....

Never run updates from recovery...

Concerning Ubuntu Server 14.04 (and possibly others)....

It's probably not something you should do.  I don't know what Ubuntu's official stance would be, but my experience has been that it's not the best thing to run apt-get dist-upgrade from a recovery kernel.  There are a few things missing, even if you mount the root partition:

• /dev, /proc, /sys and the gang are all there, but only sort of.  And /var/run?  Well, not really this go-around, if you're using 14.04 LTS.
• Upstart didn't run from this root, so there is no Upstart.  Anything that needs to deal with Upstart (like, evidently, libpam-systemd) will fail.
• If something is broken, it might get more broken if you run updates in recovery.

Still, it may become necessary.  There might even be some failures that look semi-legit but are certainly not going to help you back to a working system, because said failures are standing in the way of completing your updates.  One such issue is when one of the update scripts for one of the libraries being updated decides - suddenly - that it really needs to run a script out of /etc/init.d/.  Let's suppose, whatever the reason, no such script exists at the time of this update.  Apt fails, reporting that one of its packages wasn't able to complete all its steps.

One possible solution to this is to trick Apt into thinking there really IS a script available.  In my case, I simply:

1. touch /etc/init.d/systemd-logind
2. chmod +x /etc/init.d/systemd-logind

Viola!  Apt was able to finish processing the updates and now I might have a working system again.  I stress might because it shouldn't have broken in the first place and I have no earthly idea why it did break.  But, obviously, missing or non-configured update packages might be in the recipe.

The important thing, when you perform something abominable like what I mentioned above, is that you remove it before you reboot.  If the service does depend on Upstart, having the fake init.d script available may cause much pain.

And that's that.

Getting the LeapReader to work in Linux...sort of

I'm going to document briefly what I did, and what the problems appear to be.

TL&DR:  Remove usb_storage, blacklist it, reboot, remove it again if necessary, use VirtualBox and connect the device to a Windows VM.  Profit.

Firstly, LeapReader is a Windows-compatible thing, along with its godforsaken software.  It's NOT Linux-compatible, in any sense of the word, and let's just assume it will never be.  The developers Just Don't F-ing Care.

Not a problem, though, if you have VirtualBox to install a Windows virtual machine.  You might even be able to find some VM images from Microsoft themselves - I seem to remember them publishing images for sake of testing software against older versions.  In my case, I have a (legit) licensed copy, because unfortunately LeapFrog is not the only company that hates developing for Linux.

On the metal I'm running Linux Mint 16.  It's stocked with 3.11 for a kernel and version 204 of UDEV.  That last number will become relevant shortly.  Now, you can get LeapReader talking to the Windows VM, and VirtualBox makes that easy.  The hard part is getting LeapFrog Connect to talk to it.  And, in all honesty, I am forced to conclude it's not Connect's fault.

The problems start when UDEV and the kernel decide to try to investigate the device as a USB storage device.  Bad News.  My educated guess is that the kernel's attempts (and verbose failures, from the looks of dmesg) drive the LeapReader into a sort of temporary insanity...after which it won't talk successfully to the LeapFrog Connect program.

I tried, and tried, and tried in vain to get UDEV to stop messing around with the device.  BUT, someone somewhere removed pretty much all the commonly-known features for the commonly-published solutions.  In other words, ignore_device doesn't work, and last_rule is worthless.  At least, so far as my testing has gone.  Someone else might have better luck, and frankly I'm out of time to mess with it.

What I DID get working was to remove the usb_storage module.  I also had to blacklist it.  Strangely, Mint will load this device on boot even though it's blacklisted in modprobe.d.  At least doing an rmmod on the command line will take care of that for the current session.

It is vitally important that the device be disconnected when you remove the usb_storage module.  If usb_storage twiddles with the LeapReader, game over.  Once you are able to connect LeapReader and see nothing but basic USB interactions (no drive-style access) in the kern.log, you're good to connect the device through VirtualBox to Windows.

In the future I would very much like to make this work correctly.  Ideally, UDEV should be instructed not to fiddle with the device, and/or to represent it as anything BUT a USB storage device.  If I could even achieve the latter, I would be happy.  Unfortunately I don't know enough about the relevant subsystems to make it happen.

But, you too can now make the LeapReader work.  All you need is Windows and a whole lot of patience...and possibly an indestructible keyboard.

The Importance of umask

This is subtle and covered in countless other places.  However, for the sake of it, this is a brief illustration of umask and its (non-)interaction with the "SGID" bit.  This can be especially relevant when setting up something like a Subversion server, with user access done via SSH.

I have created a directory called test.  It's empty, and owned by me and under the group svn (my default group).

admin@svnsrv:~/test$ ls -al
total 8
drwxr-xr-x 2 admin svn 4096 May 28 15:10 .
drwxr-xr-x 3 admin svn 4096 May 28 15:10 ..

I decide to use a different group, and I set the SGID bit so that all files and directories created under "test" will get the same group ID.  I also give group write permission.

admin@svnsrv:~/test$ chgrp sshusers .
admin@svnsrv:~/test$ chmod g+ws .
admin@svnsrv:~/test$ ls -al
total 8
drwxrwsr-x 2 admin sshusers 4096 May 28 15:10 .
drwxr-xr-x 3 admin svn      4096 May 28 15:10 ..

Now we can try making another directory.

admin@svnsrv:~/test$ mkdir foo
admin@svnsrv:~/test$ ls -la
total 12
drwxrwsr-x 3 admin sshusers 4096 May 28 15:10 .
drwxr-xr-x 3 admin svn      4096 May 28 15:10 ..
drwxr-sr-x 2 admin sshusers 4096 May 28 15:10 foo

As you can see, foo inherited the correct group ID, but did not inherit group write permission.  This is where umask comes in.  We first look at our current umask, and then set a new one:

admin@svnsrv:~/test$ umask
0022
admin@svnsrv:~/test$ umask 0002
admin@svnsrv:~/test$ mkdir bar
admin@svnsrv:~/test$ ls -la
total 16
drwxrwsr-x 4 admin sshusers 4096 May 28 15:10 .
drwxr-xr-x 3 admin svn      4096 May 28 15:10 ..
drwxrwsr-x 2 admin sshusers 4096 May 28 15:10 bar
drwxr-sr-x 2 admin sshusers 4096 May 28 15:10 foo
admin@svnsrv:~/test$ umask
0002

The latest directory was created with the right permissions.  It is important to note that permissions do not inherit (to the best of my knowledge), whereas ownership does.

umask itself is designed to mask out permission bits, so it is the logical negation of the permissions we want to ultimately allow.  This does not imply that files will be created with mode 775; rather, that if a file is created with 664, a mask of 002 will retain 664 whereas a mask of 022 will reduce permissions to 644.  Likewise, a file created 700 will retain 700 with both of these example umasks.  It is, exactly, a mask: applied as a logical AND (NOT mask).  It is not a mathematical subtraction.  As an exercise to the reader, look through this Wikipedia article for more info on permissions and then work out some examples by hand, applying masks using AND (NOT mask):

http://en.wikipedia.org/wiki/File_system_permissions#Symbolic_notation

And on that note, consider this final command and its result:

admin@svnsrv:~/test$ touch baz
admin@svnsrv:~/test$ ls -la
total 16
drwxrwsr-x 4 admin sshusers 4096 May 28 15:14 .
drwxr-xr-x 3 admin svn      4096 May 28 15:10 ..
drwxrwsr-x 2 admin sshusers 4096 May 28 15:10 bar
-rw-rw-r-- 1 admin sshusers    0 May 28 15:14 baz
drwxr-sr-x 2 admin sshusers 4096 May 28 15:10 foo

See?  No execute bits on that file, even though the umask is still 0002.

N.B. The initial zero is code for "octal."  In case you didn't realize we were using octal notation here...  In the big paragraph above, the modes themselves would be properly represented as 0775 etc.

ProxMox 3.x RAID-1 Transfiguration

Proxmox 3.2 RAID-1 Conversion - GPT Mode - With Bonus Features!

The latest Proxmox can install as GPT.  Not sure if we have any say in the matter.  BUT, in case you had a die-hard need to convert your Proxmox server to RAID-1, this is a cleaned-up set of instructions.  YMMV, make sure you backup your data, check your particular partition configuration, and READ CAREFULLY.  Also it is important to note that gdisk and sgdisk will be your friend, fdisk and sfdisk will not.

Basis: http://boffblog.wordpress.com/2013/08/22/how-to-install-proxmox-ve-3-0-on-software-raid/

EDIT and RAID-5 NOTE:
This site describes some changes you need to make to get RAID-5 working.  The below is good for RAID-1 only.  The important distinctions come with the initramfs modules.  I'll try to put together another post that sums all this up....or you can just go here:

• http://terraltech.com/how-to-install-proxmox-ve-3-1-with-software-raid5-using-mdadm/#.VKMKTXVGjUY

Notes:
It is necessary to either have a valid license or to manually configure apt to use another set of repositories.  Otherwise, mdadm will not be available.  Details at:
https://pve.proxmox.com/wiki/Package_repositories

Instructions:

• Duplicate partition info onto new drive:
• sgdisk -b sda-part.txt /dev/sda
• sgdisk -l sda-part.txt /dev/sdb
• Configure partition types as RAID members:
• sgdisk -t 2:fd00 -t 3:fd00 /dev/sdb
• NOTE:  partition 1 must be type EF02 to support GRUB.  DO NOT TOUCH THIS PARTITION!
• Create RAIDs:
• boot
• mdadm --create /dev/md0 -l 1 -n 2 missing /dev/sdb2
• root
• mdadm --create /dev/md1 -l 1 -n 2 missing /dev/sdb3
• Copy /boot
• mkfs.ext3 /dev/md0
• mount /dev/md0 /mnt
• rsync -avsHAXS /boot/ /mnt/
• ## Update fstab with new boot mount: /dev/md0 (don’t use UUID)
• reboot
• Configure mdadm.conf
• mdadm -D --brief /dev/md0 >> /etc/mdadm/mdadm.conf
• mdadm -D --brief /dev/md1 >> /etc/mdadm/mdadm.conf
• Update GRUB & initramfs
• echo 'GRUB_DISABLE_LINUX_UUID=true' >> /etc/default/grub
• echo 'GRUB_PRELOAD_MODULES="raid dmraid"' >> /etc/default/grub
• echo raid1 >> /etc/modules
• echo raid1 >> /etc/initramfs-tools/modules
• grub-install /dev/sda
• grub-install /dev/sdb
• update-grub
• update-initramfs -u
• Convert BOOT partition to RAID-1
• sgdisk -t 2:fd00 /dev/sda
• mdadm --add /dev/md0 /dev/sda2
• Migrate all logical extents to /dev/md1
• pvcreate /dev/md1
• vgextend pve /dev/md1
• pvmove /dev/sda3 /dev/md1
• …. wait a long time ….
• vgreduce pve /dev/sda3
• pvremove /dev/sda3
• Convert sda3 to RAID
• sgdisk -t 3:fd00 /dev/sda
• mdadm --add /dev/md1 /dev/sda3
• …. wait a long time ….
• Reboot for final test - everything should work!
• Configure monitoring, device scans, emails, etc
• Configure SMART
• SUCCESS!!

Now you have a server that can tell you a little sooner that it might die, and have a better chance of recovering from said-death.

Why This Works:

Ordinarily, pv-moving from one partition to another of the exact same size doesn’t work, except (in our case) when the partition isn’t actually 100% in use.  As it turns out, Proxmox leaves a little extra at the end, maybe for snapshots, and consequently we can move from our original PV to one that is 1 LE smaller without issue.

Using the above technique, it should be perfectly possible to migrate to a different array configuration, such as RAID-5 or RAID-6, if you have the drives for it.  Just make sure that you load the appropriate modules into GRUB and initramfs.  All other commands should be similar, and of course there will be more of them (because there will be more drives to configure).

Why You Might Want This:

Software RAID is not officially supported by Proxmox, so you are on your own there.  BUT, if you do trust mdadm (and who doesn't?), you can have your RAID and use it, too.  There are no hardware adapters to buy, or to fail, or to curse at when they fail to read the metadata from one or more of your disks during some random reboot.  If you are very concerned about data integrity, then you should be looking at ZFS-backed iSCSI stores instead of mucking around underneath Proxmox's hood.  That said, having this all running RAID-1 does make recovery a heck of a lot easier, and especially if you couple it with a ZFS iSCSI store for all your actual VM data.

And remember:  RAID is NOT a backup solution.  If you treat it like a backup solution, you have no one but yourself to blame when all your data gets toasted after a multi-drive failure, fool!

.pyc "Bad magic number" solution

I am reporting this solution here for convenience.  Without going into specifics, the issue I had related to a Myth NetVision plugin.

I ran into an issue with a project that included only a .pyc file.  The file was compiled with Python 2.6, but the system standard Python was 2.7.  To make matters worse, several other .py files required 2.7, making reverting to 2.6 rather impossible.  The original source was not provided even in the sources; I suspect this was due to a license constraint.

The solution was to recompile the .py with 2.7.  However, to obtain the .py with going to the maintainers, I used a Python decompiler.  The following commands are what I used:

pip install unpyclib
python -m unpyclib.application -Dq path/to/file.pyc

Recompiling was as simple as moving the new source file into the directory along side the original .pyc, and deleting the .pyc before running the appropriate script (as root, due to permissions requirements).

References:
http://stackoverflow.com/questions/5287253/is-it-possible-to-decompile-a-compiled-pyc-file-into-a-py-file

Alternative Decompilers (untried):
http://stackoverflow.com/questions/5287253/is-it-possible-to-decompile-a-compiled-pyc-file-into-a-py-file
https://github.com/wibiti/uncompyle2