Quick Notes: Samba 3, CTDB, Pacemaker, and Ubuntu 12.04

I started with these links:

• http://linux-ha.org/wiki/Samba
• http://linux-ha.org/wiki/CTDB_(resource_agent)

Great stuff, but it doesn't appear to have been touched in a long time.  On Ubuntu, if you follow the path, you'll wind up broken.  Here are some intrusive fixes.

The CTDB scripts expect to find the "service" binary in /sbin.  Ubuntu 12.04 has it in /usr/sbin.  Provide a symbolic link:
  ln -s /usr/sbin/service /sbin/service

The CTDB event script 50.samba "correctly" identifies the system as a Debian one, but Samba here runs as two scripts: smbd and nmbd.  Fix the 50.samba script at the top where the switch gives us variables and what they should be.

        debian)
                CTDB_SERVICE_SMB=${CTDB_SERVICE_SMB:-smbd}
                CTDB_SERVICE_NMB=${CTDB_SERVICE_NMB:-nmbd}
                CTDB_SERVICE_WINBIND=${CTDB_SERVICE_WINBIND:-winbind}

That will get this resource working "correctly:"

primitive p_ctdb ocf:heartbeat:CTDB \
        params ctdb_recovery_lock="/opt/samba0/samba/ctdb.lock" ctdb_manages_samba="yes" ctdb_manages_winbind="yes" ctdb_start_as_disabled="yes" \
        op monitor interval="10" timeout="20" \
        op start interval="0" timeout="90" \
        op stop interval="0" timeout="100"

Of course, make sure to install winbind along with Samba3 before executing.  Watch this command:

   ctdb --socket=/var/lib/ctdb/ctdb.socket status

and the log file in /var/log/ctdb/log.ctdb while trying to start resource in case you have further problems.  I went with the special cluster IP resource, per the tutorial link at the top, so I expect that the DISABLED status of CTDB is normal; I hope I am right.  Other than making sure Samba actually starts, I've not connected any clients to it yet.  But without the above, Samba will not start.  Also, I did not find any useful resource agents for running Samba itself from Pacemaker, and attempting to use the LSB appears to break.  I wonder if it breaks for the same reason that the 50.samba script can't use it?  (50.samba calls the /etc/init.d/smbd script when it can't find /sbin/service, but it for whatever reason fail to function correctly - smbd never starts.  Invoking smbd via service appears to work fine.)

NB.  For the record, I did find one ocf script that was written for Samba on Gentoo.  It looked promising, but I didn't feel like trying to port it over to Ubuntu 12.04.

open-iscsi and pacemaker, connection issue workaroundW

I ranted previously, and much to my chagrin, for I did not look closely enough at the problem.  What I initially took as a problem with either open-iscsi or its init.d script actually appears, at the moment, to be unrelated to those things.

The problem seems non-deterministic, and I haven't pinned down exactly where the failure occurs, but here's the landscape:

We have two database server nodes attached via iSCSI to our little home-made SAN.  On each machine we have /etc/iscsi/nodes directories that are chock FULL of various targets.  However they got there, they're there, and they're not going away on their own.  Reboot as much as you like.  Now, something happens...  In my case, it was something in the combination of the SAN going completely down, one of the two attached nodes surviving the outage, and one requiring reboot (because its VM image was...well...also on the SAN).

I had learned from prior experience that when the nodes directory was loaded with junk, something fails when Pacemaker tried to reconnect the iscsi resource.  Maybe it's the resource script.  Maybe it's open-iscsi.  Who knows!   And better yet, it's not guaranteed to fail, although I noticed a lot of failures when I was testing the fencing of my nodes.  Node would go down, node would come back up, iSCSI would NOT reconnect.  Errors galore.

What I do know is that cleaning out the /etc/iscsi/nodes folder on boot tends to make this problem go away, 99.999% guaranteed.

On some clusters I have a shell script called from /etc/rc.local that kills off anything left lingering in the /etc/iscsi/{nodes,send_targets}/ folders.  Here's another way - add the following to /etc/fstab:

none /etc/iscsi/nodes ramfs defaults 0 2
none /etc/iscsi/send_targets ramfs defaults 0 2

The contents of these two folders, which appear to be relatively inconsequential (if you're not using any automatic iSCSI targets) will go away on reboot.  They don't take up much room anyway, so hopefully a ramdrive is within your budget.

Applies to Ubuntu 11.10 and 12.04.