I have a multi-terabyte, multi-server Ceph store that deadlocks itself every so often, always during high-write activity. It has been harrowing. I have bulked up on the RAM, removed slow drives from the cluster, and even disabled one of the two machines. The last case provided the best runtime, but the least reliability and of course the least space. I had to re-add the first (primary) node, since the additional space became a necessity. With the primary re-added, the deadlocks became a thing again.
It should be noted that I have CephFS mounted on the primary Ceph node. Nothing but Ceph itself is running on the secondary node. I am writing huge amounts of data onto the primary. I have another 4-server Ceph cluster that has never experienced any deadlocks, but also does NOT locally mount any Ceph device (CephFS or RBD).
The primary server had previously never had issues like this before, and originally it ran a pair of ZFS RAIDZ-5 arrays, on the same drives no less. Whatever the issue, it is likely related to Ceph (but might not be Ceph's fault). I thought perhaps the issue was with CephFS, since it's not always purported to be production-ready. I created, mapped and formatted an RBD native block device with XFS, and yet the deadlocks remained. I saw some interesting kernel tasks hung up, and searching on them brought me to where I am now.
This issue details deadlock situations and is marked as Won't Fix, since it's believed that the issue is actually one with the kernel (but not necessarily a kernel bug): http://tracker.ceph.com/issues/3076
A very lengthy thread (https://lkml.org/lkml/2004/7/26/68) discussed how a deadlock between two or more processes can and does occur, when VFS is layered on top of one or more other file systems. In my case, CephFS is on top of Ceph, which is on top of local ZFS drives. The gist of it is that a not-quite-OOM condition can occur where memory needs to allocated in order for more memory to be freed, specifically for flushing data to disk, but there is no memory available to do this. If this happens between two processes, it deadlocks without an explicit OOM, and to be fair, my servers never appear to run out of memory. On the contrary, they are almost entirely usable, except for their file systems. Issuing a SYNC (s) to sysrq-trigger before a REBOOT (b) to the same tends to preserve the logs, which are not on Ceph but are otherwise lost without the SYNC.
I tried experimenting with limiting the number of dirty pages in the system, wondering if perhaps this was part of the issue at hand. Here's the kernel reference page: https://www.kernel.org/doc/Documentation/sysctl/vm.txt. This didn't help, but what does appear to help is issuing a "3" to drop_caches. Right now, I am not sure why. But basically this is what happens: The memory stats do not indicate OOM, and the buffers/caches do not run dry. Regardless, several OSDs on the primary node cease to respond, but their processes are not dead. They all appear stuck waiting for an opportunity to write, and consume massive CPU. Ceph eventually DOWNs the OSDs, but I have noout set so they are not removed. This situation will remain thus forever, unless you hardboot....or issue a 3 to drop_caches (latest discovery). Doing the latter clears out the allocated buffers/caches, making a huge chunk available. The OSDs almost instantly begin responding again, and recovery takes place directly. Within a couple of minutes, the cluster is functioning normally again.
I suspect there is something amiss with the kernel cache system.
I have put a cron job in place to drop the caches every 2 hours, just to see how well this functions. I have done this manually a couple of times now, with good results, but this is by no means a good fix.
20171104
Logitech G920 Steering Wheel on Linux - Ubuntu 16.04, Linux Mint 18
Yes, it works. And...well, it's not terrible to make it work, but it was a bitch to find all the missing pieces.
For the search engines... think Euro Truck Simulator 2. Linux. Ubuntu and Mint. This information won't be relevant for too much longer, but for my own sanity I'm writing it down anyway.
For starters, Ubuntu 17 already has the necessary changes to the UDEV rules file. Mint 18 is based on Ubuntu 16, so you're stuck there without the following.
For the search engines... think Euro Truck Simulator 2. Linux. Ubuntu and Mint. This information won't be relevant for too much longer, but for my own sanity I'm writing it down anyway.
For starters, Ubuntu 17 already has the necessary changes to the UDEV rules file. Mint 18 is based on Ubuntu 16, so you're stuck there without the following.
Upgrading the Kernel Version
Kernel series 4.4 does not have the necessary improvements to the hid_logitech_hidpp.ko module. The 4.8 and later series kernels (supposedly also 4.6) have it. Check Linus' kernel git and locate the module source if you want to enjoy a front-row seat in the time-machine. I pushed my Mint 18 install to 4.10, partly for kicks; I figured it was a good compromise between new and working, although I have virtually no other experience and can't say what else might be fixed/broken over the 4.4 kernel that my Mint install was running...
Now, if you have a clean install of Mint, it might already be on a newer release. My work computer was on 4.8; I had done upgrades to my home PC, however, and so the kernel version seemed to lag. USE THE UPDATE MANAGER TO SELECT A NEW KERNEL VERSION!
References: http://linuxbsdos.com/2016/12/19/how-to-upgrade-the-kernel-in-linux-mint-18-1-from-4-4-to-4-8/
Also be careful /boot doesn't run out of space. That is very annoying.
Once you have the newest kernel installed, you can move on to the UDEV system and the usb_modeswitch updates.
UDEV and usb_modeswitch
It's probably a good idea to start with the usb_modeswitch. Again, newer versions of Ubuntu have this fix already loaded. You'll need to create a file in /etc/usb_modeswitch.d/ called "046d:c261". Inside, dump the following:
# Logitech G920 Racing Wheel
DefaultVendor=046d
DefaultProduct=c261
MessageEndpoint=01
ResponseEndpoint=01
TargetClass=0x03
MessageContent="0f00010142"
The next step is to insert the following rule into udev. I stuffed mine into the rules file: /lib/udev/rules.d/40-usb_modeswitch.rules, however you might want to try using the /etc/udev/rules.d/ folder instead. If the latter doesn't work, the former certainly does...you just run the slight risk of future upgrades wreaking havoc when they get to this file.
ATTR{idVendor}=="046d", ATTR{idProduct}=="c261", RUN+="usb_modeswitch '%b/%k'"
Why is this necessary?
As explained in the latter link, the steering wheel powers up in XBOX mode. A command must be sent to switch the wheel over to PC mode, or there is no PC goodness. usb_modeswitch does this.
Hazards
On these "older" versions of Ubuntu (ahem...Mint), you might run into a problem with usb_modeswitch if you're using USB hubs. Evidently the older versions of usb_modeswitch might have trouble dealing with cascaded hubs, which could explain why I was able to get this to work on my Mint PC (a laptop I plugged directly into) and not my Xubuntu desktop (where I plugged into my USB KVM switch). Newer versions of usb_modeswitch, such as that which ships with Ubuntu 17, supposedly do not suffer from this deficiency. Now, I state this only to help debug, I haven't verified it, and YMMV. Every time I pull out the steering wheel, I have to do it under cover of closed and locked door because it's a gift for my son (yeah, the kids get all the cool toys).
Also, you CAN actually get the wheel talking to Ubuntu/Mint without the newer kernel, BUT THE FORCE-FEEDBACK WILL NOT WORK! That component is contingent on the kernel module, and if it doesn't talk to the G920, then you will have nothing more than a very expensive set of potentiometers. That said, it is reassuring that to find that your wheel IS appearing to the system as a valid device of some sort, just to let you know that you're on the right track. You'll also note that the incorrect version of the kernel module will not auto-load; the correct version will. Watch lsmod and grep on "hid" to see the action.
Euro Truck Simulator 2!!!!!!
On my laptop, which is technically a work laptop, I have a shitty Intel graphics chipset, so ETS2 ran basically without textures. So that's what that looks like.... But it ran enough to let me test the wheel and get everything working without borking my wife's (significantly better) system. (Yeah, the tech guy who makes old hardware work for a long time suffers with old hardware for a long time...). Once the correct kernel module was loaded, and the udev rules were in place, I watched the syslog and observed the correct initialization sequence via usb_modeswitch occurring. On to ETS2, and enable force-feedback, and HOLY DAMN! Engine rumble, dampening, response... I even crashed on purpose to see how it would feel. Everything was solid gold.
Torcs gives some minor feedback also; I don't know much about configuring it, though, so I suppose it could be better. If anyone else has Linux racing sims they'd like to share, that support wheels and force-feedback, I'm all ears! Never thought I'd honestly buy one of these, but with the newer kernels, it is definitely a good buy.
A Side Note: the Brake Pedal
There are reports that the G29 and G920 both suffer from Logitech trying to, *cough* improve *couch* their brake pedal by inserting a rather rigid piece of rubber into the spring. You have three choices, two of which involve voiding the 1 year warranty (and mine is already like 6 months gone because the damn things has been sitting in a closet after I bought it on sale)...
- Do nothing (and keep your warranty...and possibly suffer with a very stiff brake)
- Open the device and remove the rubber insert - this reportedly returns the unit to G27 performance.
- Open the device and replace the springs with a SPRING UPGRADE!!! I totally want to do this...
This set looks pretty cool, and the reviews I have seen tend to be very positive:
The set works in the G920, and now all I need to do is to talk my wife into letting me buy it...um...for my son's steering wheel and pedal set.... ;-)
Subscribe to:
Posts (Atom)