"Quick" notes on OCFS2 + cman + pacemaker + Ubuntu 12.04

Ha ha - "quick" is funny because now this document has become huge.  The good stuff is at the end.

Getting this working is my punishment for wanting what I evidently ought not to have.

When configuring CMAN, thou shalt NOT use "sctp" at the DLM communication protocol.  ocfs2_controld.cman does not seem to be compatible with it, and will forever bork itself while trying to initialize.  This is presented as something like:

Feb 22 13:29:46 hv03 ocfs2_controld[7402]: TRY_AGAIN seen 1 times while opening checkpoint "ocfs2:controld:00000003", still tryingFeb 22 13:29:46 hv03 ocfs2_controld[7402]: TRY_AGAIN seen 2 times while opening checkpoint "ocfs2:controld:00000003", still tryingFeb 22 13:29:46 hv03 ocfs2_controld[7402]: TRY_AGAIN seen 4 times while opening checkpoint "ocfs2:controld:00000003", still tryingFeb 22 13:29:46 hv03 ocfs2_controld[7402]: TRY_AGAIN seen 8 times while opening checkpoint "ocfs2:controld:00000003", still tryingFeb 22 13:29:46 hv03 ocfs2_controld[7402]: TRY_AGAIN seen 16 times while opening checkpoint "ocfs2:controld:00000003", still tryingFeb 22 13:29:46 hv03 ocfs2_controld[7402]: TRY_AGAIN seen 32 times while opening checkpoint "ocfs2:controld:00000003"

And it goes on forever.

To make the init scripts work, some evil might be required...
In /etc/default/o2cb, add 

O2CB_STACK=cman

The /etc/init.d/o2cb script tries to start ocfs2_controld.cman before it should.  Commenting out the appropriate line makes this script do part B, which is setting up everything it should, so that CMAN can do part A and C.  OR you can try running the o2cb script AFTER cman starts, and not worrying that CMAN is "controlling" o2cb...which it really doesn't anyway.

The fact is, the cman script's main job is to call a lot of other utilities and start a bunch of daemons.  cman_tool will crank up corosync, and configure it from the /etc/cluster/cluster.conf file.  It ignores /etc/corosync/corosync.conf entirely, as proved by experimentation and documented by the cman author(s).  As far as it cares about o2cb, it runs ocfs2_controld.cman only if it finds the appropriate things configured in the configfs mount-point.  It won't find those unless you've configured them yourself or run an modified o2cb init script.

Now, it gets better.  The  /etc/default/o2cb doesn't document the cluster stack option - you have to find out by reading the o2cb init script instead.  If you let the default stack (which is, ironically, "o2cb") stand, then ocfs2_controld.cman won't run and instead complains that you're using the wrong cluster stack.  Of course, running it with the default stack then runs the default ocfs2_controld, which doesn't complain about anything at all.  But does it play nice with cman and corosync and pacemaker??

Fact is, it doesn't play with cman/corosync/pacemaker at all when it plays as an "o2cb" stack.

How is this a big deal?

The crux to all of this comes down to fencing.  OK, so suppose you have a cluster and OCFS2 configured and something terrible happens to one node.  That node gets fenced.  Then what?  Well, OCFS2 and everyone else involved should go on with life, assuming quorum is maintained.

When o2cb is configured to use the o2cb stack, it appears to operate sort of "stand-alone," meaning it doesn't seem to talk to the corosync/pacemaker/cman stack.  It doesn't get informed when a node dies, it has to find this out on its own.  Moreover, it does its own thing regardless of quorum.  Here's the thing I just did:  configure a two node cluster, configure o2cb to use the o2cb stack, and then crash one of the two nodes while the other node is doing disk access (I'm using dbench, just because it does a lot of disk access and gives latency times - great way to watch how long a recovery takes!).

Watching the log on surviving-node (s-node), you can see the o2cb stack recover based on (I assume) the timeouts configured in the /etc/default/o2cb file.  About 90 seconds later access to the OCFS2 file system is restored, regardless of the state of crashed-node (c-node).

Now the good of this is that when you start up and shut down the o2cb stack and the cman stack, they don't care about each other.  This is great because on Ubuntu these start-up and shut-down sequences seem to be all fucked up.  More about that later.  The bad news is that because these stacks are not talking, the recovery takes (my default-configured cluster) 90 seconds, which would probably nuke any VM instances running on it and wreak all sorts of havoc.  Not acceptable, and I'm not crazy about modifying defaults downward when the documentation says (and I paraphrase): "You might want to increase these values..."

Reconfigure o2cb to use the cman stack instead (O2CB_STACK=cman).  Start o2cb service, ignore the o2cb_controld.cman failure, and start the cman service.  Cman starts o2cb_controld.cman.  Update the OCFS2 cluster stack, mount and start another dbench on s-node.  Crash c-node.  This time o2cb appears to find out from the three amigos that c-node died.  However, quorum is managed by cman, and since it's a two-node cluster it halts cluster operations (such as recovery) until quorum is reestablished.  This can be done simply by restarting cman (regardless of o2cb) on c-node...once c-node is rebooted.  Unfortunately, if you're not watching your cluster crash, it could be many minutes or hours before you notice that s-node isn't able to access its data.  Or maybe never, if c-node died due to, say, releasing its magic smoke.

What else to do?  cman documentation dictates using the two_node="1" and the expected_votes="1" attributes in the cman configuration tag in /etc/cluster/cluster.conf.  Now a single node is quorate.  Let's start dbench on s-node and crash c-node again.  Recovery after c-node bites the dust takes place after about 30 seconds of downtime.  That's better.  After adding some options to configure totem for greater responsiveness (hopefully not at the cost of stability), the only thing that takes a long time now is the ocfs2 journal replay.  And that's only because my SAN is overworked and under-powered.  Donations, anyone?

Lessons Learned

To get the benefit of ocfs2 + cman + pacemaker (under Ubuntu), you need to have ocfs2_controld.cman and it has to run when "cman" is running.  That is to say, when some particular daemons - notably dlm_controld - are running.

ocfs2 can run on its own (o2cb stack), but then you lose quorum control, so to speak, and it has to be configured and managed separately of cman-and-friends.  Ugly.

For two-node clusters, make absolutely sure you have correctly configured cman to know it's a two-node cluster and expect only one vote cluster-wide, otherwise there will be no recovery for node S when node C dies.  Two node clusters under cman demand:  two_node="1" and expected_votes="1"

ocfs2_controld.cman does NOT like to talk to the DLM via sctp.  You must NOT use sctp as the communication protocol.

When configuring cluster resources, about the only things you need under this setup will be connection to the data source, and mounting of the store.  In my case, that's an iSCSI initiator resource and to mount the OCFS2 partition once I'm connected to the target.  There is NO:

• dlm_controld resource
• o2cb control resource
• gfs2 control resource

Basically, Pacemaker will not be managing any of those low-level things, unlike what you had to do back in Ubuntu 11.10.  Literally all I have in my cluster configuration is fencing, the iSCSI initiator, and the mount.  If you do anything else with the above three resources, you will find much pain when trying to put your nodes into standby or do anything with them other than leaving them running forever and ever.

Start-up sequence:
(Update 2013-02-28: The start-up order can be as now listed below.  ocfs2_controld.cman will connect to the dlm.  However, shutdown must take an alternate path.)

1. service cman start
2. service o2cb start
3. service pacemaker start

If you start o2cb first:  You can start o2cb first, but o2cb WILL complain about not being able to start ocfs2_controld.cman.  Let it complain or modify the init script to not even try, or start cman first and don't worry that cman won't try to start ocfs_controld.cman.  But you MUST use "start" and not "load" because otherwise the script will not configure the necessary attributes under configfs (/sys/kernel/config) and cman will see an o2cb-leaning ocfs2 cluster instead of a cman-leaning ocfs2 cluster.

Shutdown almost is the reverse.  Whether or not you start o2cb then cman, or cman then o2cb, you must kill cman before killing o2cb.  Sometimes on shutdown, fenced will die before cman can kill it I think and the cman init script throws an error.  Run it again ("service cman stop" - yes, again), and when it completes successfully you can do "service o2cb stop".  If you try to stop o2cb before cman is totally dead, you will wind up with a minor mess.  Given all of this, I'd recommend disabling all of these scripts from being run at system boot.

CMAN-based o2cb requires O2CB_STACK=cman in /etc/default/o2cb.

If you are upgrading from Ubuntu 11.10 to 12.04, and you want to move your ocfs2 stack from whatever it's named to cman, remember to run tunefs.ocfs2 --update-cluster-stack [target] AFTER you have o2cb properly configured and running under cman.  This will mean your whole cluster will be unable to use that particular ocfs2 store, but then if you're doing this kind of upgrade you probably should not be using it live anyway.  Since I had my resources in groups, I configured the mount to be stopped before bringing the nodes up, and allowed the iSCSI initiator to connect to the target.  Then I was able to update the stack and start the mount resource, which succeeded as expected.

I hope you find this information useful.

The Good and Bad of OCFS2

It's my own fault, really, for not having yet purchased an ethernet-controlled PDU.  I've been busy and time slips by, and the longer things run without happenstance the easier it is to forget how fragile it all is.

Whatever is causing the hiccups, it's pretty nasty when it happens.  I now have three hosts in my VM cluster.  I still run my two storage nodes, as a separate cluster.  There are two shared-storage devices, accessed by each VM host node via iSCSI, meant to distribute the load between the two storage nodes.  OCFS2 is the shared-storage file system for this installation.

Long story short, when one node dies, they all die.  15 VMs die with them, all at once.  Again, STONITH would fix this issue.  But what worries me more is the frequency of oops'.  I really can't have my VM hosts going AWOL on me just because they're tired of running load averages into the 60s.  I am beginning to rethink my design.  Here I will discuss a few pros and cons to the two approaches under consideration.

OCFS2 - Pros

• Easy to share data between systems, or have a unified store that all systems can see.
• VM images are files in directories, all named appropriately for their target VMs - No confusion, very little chance of human error.
• Storage node configuration is easy - Set up the store, initialize with OCFS2, and you're done!

OCFS2 - Cons

• Fencing is so massively important that you might as well not even use clustering without it.  Right now the cluster itself is about as stable as my "big VM host" that has motherboard and/or memory issues and regularly locks up for no apparent reason.
• You have to configure the kernel to reboot on a panic, and to panic on an oops, per the OCFS2 1.6 documentation.  I'm not really uncomfortable with that, but again the prevalence of these system failures is leaving me in wonder about the stability of everything.  I cannot necessarily pin it on OCFS2 without some better logging, or at least some hammering while watching the system monitor closely.
• One of my systems refuses to reboot on a panic, even though it says it's going to.  Don't have any idea what that's about.
• The DLM is not terrible, but sometimes I wonder how great it is in terms of performance.  I may be misusing OCFS2.  Of course, I have only one uplink per storage node to the lone gigabit switch in the setup, and the ethernet adapters are of the onboard variety.  Did I mention I need to purchase some badass PCI-e ethernet cards??

The alternative to OCFS2, when you want to talk about virtualization, is of course straight-up iSCSI.  Libvirt actually has support for this, though I'm not certain how well it works or how robust it is to failures.  However, from what I've read and seen, I'd be very willing to give it a shot.

LIBVIRT iSCSI Storage Pool - Pros

• STONITH is "less necessary" (even though it is STILL necessary) for the nodes in question, because they no longer have to worry so much about corrupting entire file systems.  They would only be at risk for corrupting a limited number of virtual machines...although, given the right circumstances I bet we could corrupt them all.
• Single node failures do not disrupt the DLM, because there is no DLM.
• iSCSI connections are on a per-machine basis, though it would be interesting to see how well this scales out.
• No shared-storage means that the storage nodes themselves can use more traditional or possibly more robust file systems, like ext4 or jfs.

LIBVIRT iSCSI Storage Pools - Cons

• Storage configuration for new and existing virtuals will require an iSCSI LUN for each one.  To keep the segregated, we could also introduce an iSCSI Target for each one, but that would become a cluster-management nightmare on the storage nodes.  It's already bad enough to think about pumping out new LUNs for the damn things.
• Since LUNs would be the thing to use, there is greater risk of human error when configuring a new virtual machine (think: Did I start the installer on the right LUN?  Hmmmm....)
• Changing to this won't necessarily solve the problems with the ethernet bottleneck.  In fact, it could very well exacerbate them.
• There is no longer a "shared storage" between machines.  No longer a place to store all data and easily migrate it from machine to machine.  At present I keep all VM configuration on the shared storage and update the hosts every so often.  This would become significantly less pleasant without shared storage.

It would probably be in my best interest to simply keep the current configuration until I can get my STONITH devices and really see how well the system stays online.  It would also behoove me to configure the VM cluster to also monitor and protect the virtuals themselves.  I tested this with one VM, but haven't done a lot to toy with all the features and functions.

So much to do, so little time.

Cascading Node Death

This might have something to do with the lack of actual STONITH devices in my configuration.  Well, it's not a "live" cluster yet, though I find the lack of stability disturbing.

It started yesterday, and culminated with a drive into the office to reboot the downed machine.  While I was there I rearranged some things, got everything back on gigabit (there was a 100 megabit switch in the mix), and added one PCI gigabit card that is only capable of about 660Mbit maximum aggregate throughput.  I need to go shopping.  Anyway, yesterday in the early afternoon, I was finally bringing some VMs online on v5.  The node had taken to a single VM instance without issue, so I decided to try migrating two others.  Then the first crack in the foundation appeared.  The node, for whatever reason, went completely dead.  The other two nodes, d1 and d2, appeared to stay up.

After the reboot and minor reconfiguration, I brought the three VMs up very slowly.  Everything seemed to go OK.  That was at about 1 am last night.  This morning, I came to my workstation to find a report that v5 was again dead.  I suspect a total kernel panic, but unfortunately without a screen attached I'll have to find out later.  To perform at least some manner of "fencing," I popped a couple of rules into IPTables to basically drop any and all traffic from v5.  This would theoretically be the same as pulling the power plug, unless there was communication below the IP layer.

Then I thought, "Hmm...maybe I can at least get things ready for some later tests."  I had been using semi-production VMs to date, mainly internal R&D stuff that isn't of much consequence.  After v5 died, I brought them back online on their original hosts.  I had set up several "sandbox" VMs on another server, and since those are most definitely NOT going to be missed by anyone, I thought I'd load them onto my iSCSI-shared storage via d2.  So around 15:00, I started an rsync to copy the VM images over.  They were cruising at about 20MB/sec.

But they never made it.

I came back to my desk to discover that d2 had died.  D1 was having a bit of trouble bringing the resources back online, so I suspected this was a case of The STONITH That Wasn't - I had been reading on some of the replies to other users of the Pacemaker mailing list that not having working STONITH can cause a hang-up (well, that's what I gathered, though it may not have been what they really said).  D2 was inaccessible, and after doing some resource cleaning I managed to get D1 to bring all cluster resources back to life.

And then D1 died.

No explanation behind it, but when I get to the office tomorrow I'm gonna beat the thing with a hammer until it tells me what the issue is.

Whilst I type this, however, there is one thing that comes to mind.  I now remember a little section of the OCFS2 installation guide, that mentions an imperative setting to basically force the kernel to reboot in the case of a particular kind of hang.  Come to think of it, I had completely forgotten it until now, and will have to see if that helps.  Naturally, that sort of thing does not seem very desirable on a virtualization host node.  I may have to rethink which file system I want to use for the shared storage.