Problems with Synology and hubs. What is the best setup for VH?

Submitted by Kokos on #1

Hello,

I have found VH a few weeks ago and after some initial testing I deployed it fully on a Synology DS3617xs. I have 25 usb2.0 license keys that i plugged into 2x usb3.0 hubs with 16 ports each. (below is a link to the hub used) Everything works when only 1 of the 2 hubs is connected. Plugging in the 2nd does not show up in VH nor in lsusb in synology ssh. What is worse, is that the Synology will hang on a reboot if both hubs are connected and prevent access to the synology. Currently I left it with 1 hub (16 usb keys) and plugged in a 17th key in the 2nd Synology port. This setup is working and reboots dont have issues.

My questions:
What could be causing this issue? The hubs being identical and maybe poorly made? or is this a problem with Synology?
Maybe I need to get 2 different models of usb hubs?
How many devices have been seen to work on Synology porducts? What are the limitations of USB controllers and VH?(My devices are all low bandwidth software security license keys)
Any suggestions for Synology? I think i can add another USB PCIe controller if needed.

Because of the reboot issue, I am worried that something like this could halt our Synology and therefore production so I am exploring alternative options for VH Servers.

What is the best case ideal setup? Is it windows/linux? Im thinking maybe to just get RPi 4 and then can scale quickly. This would require more VH licenses but it's not a problem cost wise. Would a dedicated PC with linux/windows be better? I think adding PCIe USB controllers would allow more devices per server?

I see under the "Linux USB Server" page it states "A single server can share up to 122 devices up to 6 hubs deep". Is this the best setup out of the available OSs? is this dependent on the USB controller? I'm assuming this is also true for RPi4

Any help will be greatly appreciated. I am trying to get a solid reliable server going with 25 keys and room for expansion. I doubt I will get to 50 any time soon.

Thanks,
Kokos

https://www.amazon.com/Adapter-Charging-Individual-Switches-Windows/dp/…

Submitted by Michael on Mon, 2019/10/14 - 10:33

#2

Yes its almost certainly a bug in the synology Kernel. Virtualhere will support 127 devices, but synology uses an old buggy kernel usually. e.g 3.4/3.10 or slightly better 4.4 (see here https://sourceforge.net/projects/dsgpl/files/DSM%206.2.2%20Tool%20Chain…) so ideally i would recommend a pi4 to control the dongles instead. You could just use the cloudhub image here https://www.virtualhere.com/hardware and email me and i can move your license key.

Submitted by Kokos on Sat, 2019/10/26 - 06:00

#3

So I have setup a new RPi4 and installed the armRPi4 specific build. I am having the same issue when plugging in 2x 16port hubs. When plugged in, only 12 out of 25 devices showed up. A reboot of the RPi while both hubs are connected FAILED to boot and I loose access. I have plugged in both of the USB3 hubs into the 2 USB3 ports on the RPi.
Some questions:
1. Could this be caused due to poor hub design? my guess is that the USB controller has no way of distinguishing the 2 hubs because they have the same s/n or address or whatever is normally used to distinguish different hub devices.
2. All my devices plugged in are the same. Its a low powered USB2 security key for software. I checked the data transfers and it's only a couple KB while launching the software. Is there a preference for USB 3 vs 2 hubs and/or RPi controller?

In the mean time i will be doing some tests. Like putting hub1 on USB2 and hub2 on usb3. I'll also try different hubs so that I don't have 2 of the same model.

Any advice on this will be greatly appreciated.

Thanks,
Kokos

Submitted by Kokos on Thu, 2019/10/31 - 04:47

#5

Thanks Michael.

I went ahead and ordered 3x LogiLink 10-port HUBs and will see how it goes.

Some strange things I have noticed:
- We left the RPi loaded with a full 16port hub, and added a 4 port hub. Only 1 device from the 4port hub shows up.
-Again, leaving the full 16port hub, and added 3 USB keys to the remaining 3 RPi USB ports. 19 devices showed up and have worked for 48hrs in production. Today all of a sudden about 11 Devices disappeared from VHUIclient and later the RPi became unresponsive to Ping/SSH etc. Just like the Synology was locking up and refused to boot with the 16hub + extras.
-The fully loaded 16hub by itself has not given a problem yet.

One thing that is bugging me and maybe you can elaborate on. All my USB software keys are indistinguishable when printing the VHUI xml. What I mean is the Vendor/product/idVendor/idProduct/deviceSerial is the same for every device. I am under the impression that VHUI/Hubs work with USB port addressing and this shouldn't matter but. Do you see this as a problem?

Thanks.

Submitted by Michael on Thu, 2019/10/31 - 11:39

#6

OK lets see how the new hubs go. Its ok if the devices are identical as virtualhere/linux can distinguish them by a unique address which is related to the port

Submitted by Kokos on Fri, 2019/11/08 - 06:53

#7

Hi Michael,

So the new hubs came in today. We bought 3 of the LogiLink UA0125 10port usb2.0. We hooked up 2 hubs with 10 keys in each. Everything seemed to be ok and running. Our employees started to use them and after several hours all the keys stopped working. Let me explain. The keys were visible in VHUI. I was still able to connect/disconnect devices. However, a connected device simply was not recognized anymore by the software. On all computers and all keys. A reboot of the RPi brings everything back to normal, but only temporarily. When we only had 1 hub with 16 keys plugged in, things ran smoothly without issue. Only when we started to add keys beyond a single Hub is when things start to break. We also tried 1x16 hub + 3 more keys in the remaining 3 RPi ports for a total of 19. This also only worked for a few hours.

The way i installed was to flash the RPi with latest factory image and then i installed the RPi4 optimized VHUI server. Haven't touched anything else.

Do you have any suggestions? Logs/Files I can check before/after the issue?
I'm not sure where to go from here. I'd hate to have to buy an RPi per 1x10 hub, but that might have to be the solution if nothing else works.

Thanks

Submitted by Michael on Fri, 2019/11/08 - 09:32

#8

OK thats a surprise. The syslog will say why they all dropped . Can you do this

grep SURPRISE /var/log/syslog on the server and see if there is anything

In the meantime i think you will need one pi4 per hub

Submitted by Kokos on Sat, 2020/01/18 - 06:54

#9

Hi Michael,

We finally got around to doing some more detailed tests. With 3 hubs, the RPi restarted itself after 30min. Dmesg returns the following:
[ 541.121583] usb 1-1.2.5: reset full-speed USB device number 20 using xhci_hcd
[ 541.551537] usb 1-1.2.5: reset full-speed USB device number 20 using xhci_hcd
[ 1013.971743] usb 1-1.4.2: reset full-speed USB device number 12 using xhci_hcd
[ 1014.391538] usb 1-1.4.2: reset full-speed USB device number 12 using xhci_hcd

The syslog you referred to is non existent on our system. We currently have 2 RPi, 1 with a normal OS + RPi4-install and the other we used the VHUI Cloud Image.

After a short while of not touching anything, we had a list of further errors:

[ 1992.764335] xhci_hcd 0000:01:00.0: WARN Event TRB for slot 28 ep 3 with no TDs queued?
[ 1992.772280] xhci_hcd 0000:01:00.0: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state.
[ 2034.872073] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2035.912050] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2036.951836] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2038.001842] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2039.032029] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2039.037359] usb 1-1.4.4-port2: Cannot enable. Maybe the USB cable is bad?
[ 2040.071834] usb 1-1.4.4-port2: cannot disable (err = -110)
[ 2041.112033] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2042.152022] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2043.192069] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2044.231863] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2045.271969] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2045.277299] usb 1-1.4.4-port2: Cannot enable. Maybe the USB cable is bad?
[ 2046.312061] usb 1-1.4.4-port2: cannot disable (err = -110)
[ 2047.352026] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2048.391989] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2049.431839] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2050.471843] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2051.521864] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2051.527195] usb 1-1.4.4-port2: Cannot enable. Maybe the USB cable is bad?
[ 2052.551853] usb 1-1.4.4-port2: cannot disable (err = -110)
[ 2053.591949] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2054.631870] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2055.671950] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2056.711889] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2057.752028] usb 1-1.4.4-port2: cannot reset (err = -110)
[ 2057.757357] usb 1-1.4.4-port2: Cannot enable. Maybe the USB cable is bad?
[ 2058.791811] usb 1-1.4.4-port2: cannot disable (err = -110)
[ 2059.832038] usb 1-1.4.4-port2: cannot disable (err = -110)
[ 2060.872392] usb 1-1.2-port3: cannot reset (err = -110)
[ 2061.912247] usb 1-1.2-port3: cannot reset (err = -110)
[ 2062.952078] usb 1-1.2-port3: cannot reset (err = -110)
[ 2063.992327] usb 1-1.2-port3: cannot reset (err = -110)
[ 2065.032077] hub 1-1.4.4:1.0: hub_ext_port_status failed (err = -110)

Suspecting something to do with the usb3.0 port, we plugged in the 3 hubs in series to the USB2.0 port on the Pi. Following errors occurred.

[ 27.571627] usb 1-1.4.4.3: reset full-speed USB device number 24 using xhci_hcd
[ 28.001547] usb 1-1.4.4.3: reset full-speed USB device number 24 using xhci_hcd
[ 35.591588] usb 1-1.3.4.1.5: reset full-speed USB device number 27 using xhci_hcd
[ 36.031557] usb 1-1.3.4.1.5: reset full-speed USB device number 27 using xhci_hcd
[ 52.711582] usb 1-1.4.7: reset full-speed USB device number 23 using xhci_hcd
[ 53.141559] usb 1-1.4.7: reset full-speed USB device number 23 using xhci_hcd

All our devices/hubs are USB2.0, would it make sense to somehow disable USB3.0 (xhci) or maybe you have a better suggestion?

After the errors start ALL keys stop working (remember they are all the same type for the same software). So things work for about 30-60min before all devices become unusable.

Please let me know if there is anything we can try. I'd really like to have this working stable like the SEH product we used previously. It would be a shame if we had to continue to purchase their expensive hardware for every 20 USB keys.

Thanks in advance

Submitted by Michael on Sat, 2020/01/18 - 17:16

#10

OK this is unforunate "Event TRB for slot 28 ep 3 with no TDs queued?" looks like there is still a bug in the kernel

I will investigate this next week and let you know when i have more information...

Submitted by Michael on Tue, 2020/01/21 - 11:19

#11

Plug all the hubs into the USB 2 ports for the time being and see if that resolves the issue

Submitted by Kokos on Fri, 2020/01/24 - 06:47

#12

We have already tried (i mentioned it in previous post). It seems that it is still using the xhci_hcd driver which i believe to be 3.0. Do you know of a way to completely disable/remove the 3.0 driver? some type of manual work around to specify a 2.0 driver?

Submitted by Michael on Fri, 2020/01/24 - 11:37

#13

I just looked up the schematic of the pi4 and even the usb 2 ports are on the xhci controller, i didnt realize this. They all connect over the PCIe interface of the Pi4 BCM2711 chip. Therefore the kernel XHCI driver must be used there is no downgrade to just usb2 ehci.

I dont think there is an xhci fix for this at the moment, could i suggest trying a pi3b (not the plus)

Submitted by Kokos on Mon, 2020/02/17 - 10:45

#14

Looks like the pi3b is not the right solution here. It only works for <10 usb keys. Once we start adding a second 10-port hub, the devices fail to show up in VirtualHere. Playing around with plugging the keys in/out has random results with them showing up and disappearing but in the end it seems like the limit is consistently 10. So far, at least we don't have the crashing problem where all keys would fail while being used.

I'm not sure where to go from here or where the problem is. Is it RaspberryPi, VirtualHere, or USB-devices causing these issues. Would it make sense to use a PC with a linux install and use the the onboard and/or PCI USB add-in cards? I am really trying to hang in here as this solution is perfect...if only it worked reliably.

Thanks,
Kamil

Submitted by Michael on Mon, 2020/02/17 - 21:34

#15

Could you email me the syslog and dmesg from the pi3b (if its still available) after it failed with +10 devices. I want to see what the kernel is saying. Thanks

My guess is its still a kernel issue. e.g https://www.mail-archive.com/linux-usb [at] vger.kernel.org/msg113780.html

I think you might have to use the synology for 16 and the pi for 10 devices , since that is stable. The client can connect to multiple servers at once so end users wont be too affected.

Submitted by Michael on Tue, 2020/03/17 - 18:48

#17

I would recommend useing the synology for 16 and the pi for 10 devices , since that is stable. The client can connect to multiple servers at once so end users wont be too affected.

Submitted by Kokos on Thu, 2020/03/19 - 15:47

#18

Actually, they are not that stable. At first it seemed like everything was good with 10 devices in a Pi. However, upon a reboot, only 7 devices were showing. Both the Synology and Pi devices are finicky when starting to fill up a hub. Doing so one-by-one seems to get the most working devices. But again upon a reboot, some devices randomly disappear from the list. I can't pinpoint anything specific because if i reboot the Pi 5 times in a row with ten devices, i can expect a different amount between 7-10 devices to show up each time.

Am I in a unique situation with this type of setup? In your testing have you had Pi's working with large amounts of devices? I'd like to try something that is known to be working. Maybe different OS?

Submitted by Kokos on Wed, 2020/03/25 - 08:13

#20

I did some testing today. First I started by re-flashing all 3 Pi's (2x RPi4 and 1x RPi3) with the latest CloudHub 4.1.1. After booting and connecting various amounts of hubs and usb keys, I have some findings.

First, the RPi3. It was loosing keys (key = usb2.0 license dongle) randomly. Upon further inspection, the ports on the Pi3 seem to be loose and wiggle causing an unstable connection. For now we left it alone with nothing plugged in.

RPi4. When connecting keys and hubs, it seemed pretty normal. We tried and were able to get 17 Keys to show up. I have noticed port assignment failures in the log, but it looks like after 3-5min of retrying it stabilized. During the first 1-5min after plugging keys/hubs, the keys sporadically show/disappear respectively to the logs.

Currently, we left 12 keys and 10 keys in the RPi4s. All keys can be used and so far no issues after initial connection. Hopefully during production there will be no issues and we can finally get some success. finger crossed.

FIRMWARE: None of the commands work from the link you provided. Im guessing this is because I have the Cloud version installed. Anything I should do here? maybe go with default Raspbian and install VHUI the traditional way?

Thanks

Submitted by Michael on Wed, 2020/03/25 - 10:30

#21

Ah OK I thought you were using raspbian, so cloudhub wont have the firmware update commands, only raspbian does