The Good and Bad of OCFS2

It's my own fault, really, for not having yet purchased an ethernet-controlled PDU.  I've been busy and time slips by, and the longer things run without happenstance the easier it is to forget how fragile it all is.

Whatever is causing the hiccups, it's pretty nasty when it happens.  I now have three hosts in my VM cluster.  I still run my two storage nodes, as a separate cluster.  There are two shared-storage devices, accessed by each VM host node via iSCSI, meant to distribute the load between the two storage nodes.  OCFS2 is the shared-storage file system for this installation.

Long story short, when one node dies, they all die.  15 VMs die with them, all at once.  Again, STONITH would fix this issue.  But what worries me more is the frequency of oops'.  I really can't have my VM hosts going AWOL on me just because they're tired of running load averages into the 60s.  I am beginning to rethink my design.  Here I will discuss a few pros and cons to the two approaches under consideration.

OCFS2 - Pros

• Easy to share data between systems, or have a unified store that all systems can see.
• VM images are files in directories, all named appropriately for their target VMs - No confusion, very little chance of human error.
• Storage node configuration is easy - Set up the store, initialize with OCFS2, and you're done!

OCFS2 - Cons

• Fencing is so massively important that you might as well not even use clustering without it.  Right now the cluster itself is about as stable as my "big VM host" that has motherboard and/or memory issues and regularly locks up for no apparent reason.
• You have to configure the kernel to reboot on a panic, and to panic on an oops, per the OCFS2 1.6 documentation.  I'm not really uncomfortable with that, but again the prevalence of these system failures is leaving me in wonder about the stability of everything.  I cannot necessarily pin it on OCFS2 without some better logging, or at least some hammering while watching the system monitor closely.
• One of my systems refuses to reboot on a panic, even though it says it's going to.  Don't have any idea what that's about.
• The DLM is not terrible, but sometimes I wonder how great it is in terms of performance.  I may be misusing OCFS2.  Of course, I have only one uplink per storage node to the lone gigabit switch in the setup, and the ethernet adapters are of the onboard variety.  Did I mention I need to purchase some badass PCI-e ethernet cards??

The alternative to OCFS2, when you want to talk about virtualization, is of course straight-up iSCSI.  Libvirt actually has support for this, though I'm not certain how well it works or how robust it is to failures.  However, from what I've read and seen, I'd be very willing to give it a shot.

LIBVIRT iSCSI Storage Pool - Pros

• STONITH is "less necessary" (even though it is STILL necessary) for the nodes in question, because they no longer have to worry so much about corrupting entire file systems.  They would only be at risk for corrupting a limited number of virtual machines...although, given the right circumstances I bet we could corrupt them all.
• Single node failures do not disrupt the DLM, because there is no DLM.
• iSCSI connections are on a per-machine basis, though it would be interesting to see how well this scales out.
• No shared-storage means that the storage nodes themselves can use more traditional or possibly more robust file systems, like ext4 or jfs.

LIBVIRT iSCSI Storage Pools - Cons

• Storage configuration for new and existing virtuals will require an iSCSI LUN for each one.  To keep the segregated, we could also introduce an iSCSI Target for each one, but that would become a cluster-management nightmare on the storage nodes.  It's already bad enough to think about pumping out new LUNs for the damn things.
• Since LUNs would be the thing to use, there is greater risk of human error when configuring a new virtual machine (think: Did I start the installer on the right LUN?  Hmmmm....)
• Changing to this won't necessarily solve the problems with the ethernet bottleneck.  In fact, it could very well exacerbate them.
• There is no longer a "shared storage" between machines.  No longer a place to store all data and easily migrate it from machine to machine.  At present I keep all VM configuration on the shared storage and update the hosts every so often.  This would become significantly less pleasant without shared storage.

It would probably be in my best interest to simply keep the current configuration until I can get my STONITH devices and really see how well the system stays online.  It would also behoove me to configure the VM cluster to also monitor and protect the virtuals themselves.  I tested this with one VM, but haven't done a lot to toy with all the features and functions.

So much to do, so little time.

Tidbits - Live Migration, Bonding, Bridging, DNS

Live Migration - NEVER TURN THEM OFF!!

I today succeeded in performing a live migration of a VM, from one host to another.  From the source host:

virsh migrate --live  --verbose --domain myguest --desturi qemu+ssh://newhost/system

It took all of about 8 seconds to complete the migration.  No downtime.  It is said this is done in such a way that connections are not even interrupted.  I would like to test that for myself. :-)

Bonding, Bridging, and DNS

Ubuntu 12.04 now requires that DNS resolver directives reside in the /etc/network/interfaces file, or else are placed somewhere I haven't found out about yet.  Here are the key directives, which should go with the adapter definition (as in, under iface eth0 inet static):

dns-search my.local.domain my.other.local.domain
dns-nameservers 192.168.4.4 192.168.4.5

For 10.04 up, bonding and bridging seem to be a lot easier.  Unfortunately, the documentation for these features SUCKS (meaning it's almost non-existent, and forget the configuration examples, too).  Here I configure a two-nic bond, using mode 6 (balance-alb - no switch support required).  I also stuff the bond into a bridge (br0), since this host is also serving VMs:

auto eth0 eth1 br0 bond0 

iface eth0 inet manual
  bond-master bond0 

iface eth1 inet manual
  bond-master bond0 

iface bond0 inet manual
  bond-miimon 100
  bond-slaves none
  bond-mode   6 

iface br0 inet static
  bridge-ports bond0
  address 10.17.0.124
  netmask 255.240.0.0
  gateway 10.16.0.1
  bridge-stp on
  bridge-fd 0
  bridge-maxwait 0

Some things to note about the above:

• bond-slaves none was the recommendation of the forums and docs - the slaves are defined by specifying their master for each interface.  This seems to work rather well.
• bond-mode now suggests it's even easier to set up different kinds of bonds on the same machine.  Previously you had to do this by aliasing the bonding driver multiple times with different options.  Not terrible, just a little more clunky.
• The bridge-* options used to be bridge_*.  Note the dash is used instead of the underscore.  Moreover, the forward-delay (fd) and maxwait are set to zero here.  Trying to set them manually or via a method other than this seems to fail terribly - they'll always go back to their defaults!  This is the first time I've been able to get my bridge to not have the 15 second forwarding delay on new interfaces, though I am anxious to actually validate it beyond any doubt.  The configuration tells me that's the way it is, so hopefully it tells the truth.

I tied the VM to the bridge in the NIC configuration of the VM definition (libvirt-style):

   

     

     

     

   

Note: Remember that when creating new VMs, just omit the

and the tags, and they will be auto-filled with appropriate or auto-generated values.

Cascading Node Death

This might have something to do with the lack of actual STONITH devices in my configuration.  Well, it's not a "live" cluster yet, though I find the lack of stability disturbing.

It started yesterday, and culminated with a drive into the office to reboot the downed machine.  While I was there I rearranged some things, got everything back on gigabit (there was a 100 megabit switch in the mix), and added one PCI gigabit card that is only capable of about 660Mbit maximum aggregate throughput.  I need to go shopping.  Anyway, yesterday in the early afternoon, I was finally bringing some VMs online on v5.  The node had taken to a single VM instance without issue, so I decided to try migrating two others.  Then the first crack in the foundation appeared.  The node, for whatever reason, went completely dead.  The other two nodes, d1 and d2, appeared to stay up.

After the reboot and minor reconfiguration, I brought the three VMs up very slowly.  Everything seemed to go OK.  That was at about 1 am last night.  This morning, I came to my workstation to find a report that v5 was again dead.  I suspect a total kernel panic, but unfortunately without a screen attached I'll have to find out later.  To perform at least some manner of "fencing," I popped a couple of rules into IPTables to basically drop any and all traffic from v5.  This would theoretically be the same as pulling the power plug, unless there was communication below the IP layer.

Then I thought, "Hmm...maybe I can at least get things ready for some later tests."  I had been using semi-production VMs to date, mainly internal R&D stuff that isn't of much consequence.  After v5 died, I brought them back online on their original hosts.  I had set up several "sandbox" VMs on another server, and since those are most definitely NOT going to be missed by anyone, I thought I'd load them onto my iSCSI-shared storage via d2.  So around 15:00, I started an rsync to copy the VM images over.  They were cruising at about 20MB/sec.

But they never made it.

I came back to my desk to discover that d2 had died.  D1 was having a bit of trouble bringing the resources back online, so I suspected this was a case of The STONITH That Wasn't - I had been reading on some of the replies to other users of the Pacemaker mailing list that not having working STONITH can cause a hang-up (well, that's what I gathered, though it may not have been what they really said).  D2 was inaccessible, and after doing some resource cleaning I managed to get D1 to bring all cluster resources back to life.

And then D1 died.

No explanation behind it, but when I get to the office tomorrow I'm gonna beat the thing with a hammer until it tells me what the issue is.

Whilst I type this, however, there is one thing that comes to mind.  I now remember a little section of the OCFS2 installation guide, that mentions an imperative setting to basically force the kernel to reboot in the case of a particular kind of hang.  Come to think of it, I had completely forgotten it until now, and will have to see if that helps.  Naturally, that sort of thing does not seem very desirable on a virtualization host node.  I may have to rethink which file system I want to use for the shared storage.