I've concluded my research and here is the answer.
It's dead, Jim.
If your virtual's MACs are getting squashed, look no further than Balance-ALB (mode 6).
I'm not sure if this can be fixed or not, but right now it sucks. After much testing and lots more reading, it seems this is a "known problem" and doesn't look like it's going to be fixed. For reference, here's the configuration:
vnet0 -> br0 -> bond0 -> eth1, eth2, ... (note to self: make this a pretty picture)
Where bond0 is mode-6 over the listed interfaces. ALB is supposed to balance transmits AND receives, so to accomplish this it apparently snags ARPs from the wire and replaces them with one of the several MACs of its slaves. I think, if I recall correctly, it picks a slave in a round-robin fashion. Anyway, the problem seems to be that when ARPs for the virtuals under the bridge come in, ALB snags those as well and scarfs them, stomps them, and sends out its own MAC.
Thus, the spice does not flow.
Symptoms include: intermittent ping, intermittent connectivity, ARP table reading one of the bond's MACs instead of the virtual's, headache, nausea, and some minor vomiting.
Nothing else, save mode-0 (which really doesn't count) does any sort of receive-side load-balancing. What I would REALLY like to see is ALB more intelligently handle ARP requests, such that it doesn't squash those of the virtuals that are properly serviced elsewhere. Ideally, it should not squash any ARP replies that do not have anything to do with its own physical adapters. That seems like it ought to be a relatively easy fix...except I don't know where in the code to fix it...yet. I might try someday...
Use Mode-4 you say? Nope, it doesn't load-balance under the bonding driver. Check out libteam instead and then go crazy trying to build it under Ubuntu 11.10. It looks like it's under 12.04, except my stupid cluster doesn't run 12.04 because of issues with OCFS2 and the DLM and a lot of other bullshit that is taking way too much energy to solve. Plus, Mode-4 is great if you're using a single switch. I'm using two, for redundancy. I could go back to Sins of the Bond and team two mode-4's under maybe a mode-1 (XOR), but then I'm kinda back where I started with no really awesome receive-load-balancing.
The ALB problem is especially nefarious because occasionally the virtual's REAL MAC will appear in the ARP cache. Now, just What The Fuck is up with that? It makes me think this issue with ALB is more of a bug than a feature.
Temporary fixes in the meantime: switch to TLB (mode 5), or any other mode that doesn't involve borking ARPs. Strangely, even though TLB manages load-balancing on transmit, it doesn't display the same ARP-hell that ALB suffers.
No comments:
Post a Comment