Linux: client hangs with apparent kernel panic on armhf (tested on rpi2/3 and Orange Pi PC)

I'm considering VirtualHere software as a solution for the "home grown" VDI I've got under my control. The solution we implemented is based on mostly OSS components (libvirt et al) but the question of user-friendly USB over IP sharing remains to be covered. For VDI we use raspberry pi 2/3 as a thin clients but end users are not limited by VDI - some people prefer utilizing LXDE or Mate desktops as shipped with raspbian (or armbian for few Orange Pi PC's we have).

One of the key parts of typical day-to-day end-user activity is a usage of a shared network scanner with auto-feed unit installed. It is actually a bit Canon copier that have got an USB interface and it works sufficiently well with saned. Current approach for end-users to use this unit is to RDP, x2go or VNC into a dedicated RPi2 which is in turn connected to this copier using USB cable. What I want is to move from this remote-access scheme into using USB-over-IP sharing solution and that's where VirtualHere seems to be a silver bullet.

Before buying licenses I decided to to some testing at home using my Win 10 PC as a server and RPi2 with Retropie installed as a client. On windows side I had downloaded and started up x64 version of VH server, on RPi side I downloaded and executed vhclientarmhf linux client command line tool. As I do not use avahi/bonjour/zero-conf at my home LAN I had to manually add remote USB HUB on RPi side using "vhclientarmhf -t "MANUAL HUB ADD,". Then thing seemed to start working as they should - I've been able to list remote USB devices and issue USE command for one of them. Things were looking good for ~5 minutes but then RPi stopped responding over the network. So I went into living room (where RPi resides serving as a retro game console for children to play on) to check if RPi responds to attached keyboard, gamepads or over HDMI-CEC emulated input device. None were functioning. Screen output "froze" at what it was the moment RPi stopped responding. All in all the behaviour seems to be a typical kernel panic but without any debug output provided on console.

So I tried the same on another RPi I have at home, RPi3 with OSMC installed to be used as a Kodi mediacenter box. Things worked the same way: remote input device (I've been using a USB gamepad as a testing equipment) worked OK for several minutes and then RPi simply hanged with no debug output either over network or to the attached HDMI screen. Next to try was Orange Pi PC I've got on hand serving as a home VOIP gateway. It have got different version of linux kernel (I've tested on 3.4.113-sun8i) and have got armbian installed instead of raspbian-based distros I've got on RPis. The behaviour was exactly the same though - some time things worked well and then the system came to a complete hang.

Is it a known issue? Am I doing something wrong to cause kernel panics on RPis and OPiPC? Any suggestions on how to debug and fix this issue? Thanks in advance.

P.S. I'm a UNIX devops engineer so I wouldn't be frighten to do some stuff like recompiling kernel, configuring netconsole, e.t.c. What I'm limited with is a free time so I'm seeking for a good suggestion/assistance on debugging this one so I won't waste a lot of precious time on fiddling in a wrong way.

#2

seems like the interrupts are actually being disabled and never reenabled. This was a bug in the usbip linux kernel code about a year ago and was fixed here ( https://github.com/torvalds/linux/commit/21619792d1eca7e772ca190ba68588… ) .

Not sure what kernel you are running on your pi but try running the latest like this sudo BRANCH=next rpi-update does that fix the problem now?

#3

Hi Michael, thanks for providing this info.

rpi-update is not relevant for osmc (it uses its own kernel build which does not include usbip stuff so I've been compiling it myself for that kernel). OSMC currently is on 4.4.27-7-osmc kernel version. On retropie kernel version 4.4.26-v7+ is being used. Both are pretty old compared to the fix version for the patch you mentioned. I would check sources I used for OSMC for this patch to be applied and report back.

As for Orange Pi PC - it uses sunxi kernel branch which is currently at 3.4.113-sun8i. I'm not sure if patch had been backported to this kernel (and if it is applicable at all to such an old version). Would also try to check this after dealing with RPi-related stuff.

#4

Well, it turned out that rpi-update is not is not available in default install of the OS. So I installed it with apt and dug into rpi-update sources.

Looking at the contents of the git repo it uses to update kernel BRANCH=next part is no longer required - 3 days ago kernel version for master branch had also been bumped to 4.9 branch. Updating kernel to latest available at default (master) branch fixes the issue, no more "kernel stuck" behavior observed after several hours of testing.

What it means is a fact that stock install of raspbian-based distros is not compatible with VH usb client. IMO it should be stated with a big letters on linux client download page together with the info on how to workaround the problem (as long as user is OK with updating kernel from external - i.e. non distro - source).

I assume that problem is the same for OSMC and Orange Pi PC. I would build up required kernel module myself for these with irq patch included and report back in case it won't fix the issue. Unless there still would be some issues it is OK to mark this thread as [SOLVED]. MIchael, thanks once again for your help.

#5

OK good glad its fixed, ill update the download page...