SURPRISE UNBOUND Every Few Seconds

Submitted by Kirilian on #1

I have come across a particular USB device that exhibits some strange behavior. I am running the server on a Raspberry Pi 4. The client is set to Auto Use All Devices on the server. Here is what I am seeing...

1) When the device is connected, it immediately appears in the device list and becomes "In Use". In a few seconds, the device disappears (along with the Windows sound indicating a device has been unplugged). In about a second or two later, the device re-appears for a few seconds then disappears. The cycle continues...

2) If I set Auto Use All Devices to OFF, the device is stable in the device list (it does not come and go). If I then attempt to Use This Device, it disappears in a few seconds and then reappears in the device list (without the In Use).

3) When Auto Use All Devices is on, and the device is repeatedly cycling, the device's user interface software on Windows has no problem connecting to it. Once connected, the cycling stops. The device remains connected and In Use.

Here are just a couple of entries from the syslog. This repeats over and over

Jan 2 00:44:54 CableBlaster vhusbdarm[456]: Device 113 [20a9:0065] BOUND to connection 83
Jan 2 00:45:08 CableBlaster vhusbdarm[456]: Device 113 [20a9:0065] SURPRISE UNBOUND from connection 83
Jan 2 00:45:08 CableBlaster vhusbdarm[456]: Unmanaging device 113 [20a9:0065]
Jan 2 00:45:11 CableBlaster vhusbdarm[456]: Found Full speed device [20a9:0065] "Autotronic Controls Inc , MSD Virtual COM Port " at address 113
Jan 2 00:45:11 CableBlaster vhusbdarm[456]: Device 113 [20a9:0065] BOUND to connection 83
Jan 2 00:45:12 CableBlaster vhusbdarm[456]: Device 113 [20a9:0065] SURPRISE UNBOUND from connection 83
Jan 2 00:45:12 CableBlaster vhusbdarm[456]: Unmanaging device 113 [20a9:0065]
Jan 2 00:45:14 CableBlaster vhusbdarm[456]: Found Full speed device [20a9:0065] "Autotronic Controls Inc , MSD Virtual COM Port " at address 113

I fully believe this is NOT a problem with VirtualHere since other (different) devices work just fine on the server. I have access to several of these problem 'devices', and they all behave the same way. I have tried this on multiple different Pis, as well as multiple different PCs running the VirtualHere client, and the results are the same. I have no idea what this manufacturer is doing when the device is not communicating with it's user interface, but it doesn't appear to be normal.

Also, the device does not behave this way when connected directly to the PC.

Any thoughts on what could cause SURPRISE UNBOUND only when In Use AND not communicating with any software, yet is stable when communicating or Not In Use?

Submitted by Michael on Thu, 2020/01/09 - 17:53

#2

Some devices (not many) dont like "RESET" commands sent to them once they have received that command once. This can be resolved by doing this (dont use the device via virtualhere yet)

right click on the device and select "Custom Event Handler...." and paste in exactly this line:

onReset.$VENDOR_ID$.$PRODUCT_ID$=

then press OK. Now use the device via VirtualHere and see if it works.

This setting is remembered (which skips a reset command) and so you only have to do this once.

Submitted by Kirilian on Fri, 2020/01/10 - 11:51

#3

Ok, here are the steps I tried...

1) Disable Auto Use All Devices
2) Problem device appears in device list (stable - not connecting/disconnecting periodically)
3) Right-clicked on the problem device and selected "Custom Event Handler..."
4) Copied and pasted the line above into the box. Ensured no extra spaces at the beginning/end, etc.
5) Pressed OK
6) Right-clicked on the problem device and selected Use This Device
7) Device transitioned to In Use for a few seconds, t hen disappeared and then re-appeared as Not In Use

It almost seems as if the device is attempting to find its user interface software on the PC side, but when unable, creates an error condition that gets interpreted as an unplug. Even though the device is cycling between In Use and then disappearing, it still connects with the software as soon as it is opened.

Thank you for your help!

Submitted by Michael on Fri, 2020/01/10 - 12:41

#4

A SURPRISE UNBOUND means one of two things

1. The user has physically unplugged the device while its "in-use" via virtualhere

2. The device is disconnecting itself internally from the USB bus (probably an internal device firmware reset). USB devices, especially complex ones are themselves little computers which can reboot etc just like a normal full size computer and they will drop off and reappear on the bus when the do this.

You scenario is item 2, the device itself is disconnecting and reconnecting, its not related to virtualhere.

on the pi4, have a look a dmesg and see if that logs any information when it drops off the usb bus and comes back on.

Submitted by Kirilian on Thu, 2020/01/16 - 15:17

#5

Ok, I gave dmesg a try. At first I wasn't really seeing anything. The device seemed to be connected and the system was stable. Then I set VirtualHere client to Auto Use Device and then refreshed dmesg. Below is what repeated numerous times until I disabled Auto Use Device...

usb 1-1.4: new full-speed USB device number 6 using xchi_hcd
usb 1-1.4: New USB device found, idVendor=20a9, idProduct=0065, bcdDevice= 2.00
usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1.4: Product: MSD Virtual COM Port
usb 1-1.4: Manufacturer: Autotronic Controls Inc
usb 1-1.4: SerialNumber: 4y04w372XC
usb 1-1.4: reset full-speed USB device number 6 using xchi_hcd
usb 1-1.4: reset full-speed USB device number 6 using xchi_hcd
usb 1-1.4: USB disconnect, device number 6
usb 1-1.4: new full-speed USB device number 7 using xchi_hcd
...repeat

Submitted by Michael on Thu, 2020/01/16 - 17:59

#6

OK unfortunately dmesg is not providing any more logging information, i dont think this device is going to work via virtualhere

Submitted by MarkoD on Fri, 2020/01/17 - 11:43

#7

Sorry for jumping in - we've had the SURPRISE UNBOUND issue happen with one CloudHub unit (running on GL-MT300N-V2) out of four. It happenens just about once a week:

Thu Jan 16 12:14:34 2020 kern.err kernel: [689747.048504] usb usb2-port1: disabled by hub (EMI?), re-enabling...
Thu Jan 16 12:14:34 2020 kern.info kernel: [689747.054945] usb 2-1: USB disconnect, device number 3
Thu Jan 16 12:14:34 2020 user.info vhusbdchglmt300nv2[986]: Device 21 [10c4:ea60] SURPRISE UNBOUND from connection 21
Thu Jan 16 12:14:34 2020 user.info vhusbdchglmt300nv2[986]: Unmanaging device 21 [10c4:ea60]
Thu Jan 16 12:14:35 2020 kern.info kernel: [689747.492205] usb 2-1: new full-speed USB device number 4 using ohci-platform
Thu Jan 16 12:14:35 2020 user.info vhusbdchglmt300nv2[986]: Found Full speed device [10c4:ea60] "Silicon Labs, CP2102N USB to UART Bridge Controller" at address 21
Thu Jan 16 12:14:35 2020 kern.info kernel: [689747.992201] usb 2-1: reset full-speed USB device number 4 using ohci-platform
Thu Jan 16 12:14:36 2020 user.info vhusbdchglmt300nv2[986]: Device 21 [10c4:ea60] BOUND to connection 21

The USB device is connected directly to the CloudHub, without any USB hubs. It may be the USB device malfunctioning (and resetting by itself) or perhaps a power issue?

PS: Would it be possible to use the event system to run a script that would send an email when this happens?

Submitted by Michael on Fri, 2020/01/17 - 13:13

#8

usb usb2-port1: disabled by hub (EMI?), re-enabling...

That seems to be the culprit as it kicks the device off the hub and then its reenumerated. Try a powered hub (ideally) or just any hub between the cloudhub and the usb device and see if it still happens

Regarding the event script actually yes the server can call it when a "SURPRISE UNBOUND" happens.

e.g onUnbind=/root/script.sh "$DEVPATH$" "$SURPRISE_UNBOUND$"

$SURPRISE_UNBOUND$ will be set to 1 when this occurs

Submitted by MarkoD on Mon, 2020/01/20 - 01:16

#9

Thanks for the instructions! If I want to also power cycle the port (via command power_cycle_port I guess?), how can I combine this with the email script?

Submitted by Michael on Mon, 2020/01/20 - 10:13

#10

You can do this but you will need to mix in the virtualhere client like this:

Where you run the onUnbind script you will also need to run the virtualhere client. So this is your pi.

There is a virtualhere client for the pi at https://www.virtualhere.com/usb_client_software

That needs to run in the background as a daemon.

Then your unbind script needs to call vhclientarmhf -t "CUSTOM EVENT,address,power_cycle_port"

You will actually need to change the onUnbind line in the server config.ini so it passes in the address like this

onUnbind=/root/unbind.sh "$ADDRESS$" "$DEVPATH$" "$SURPRISE_UNBOUND$"

Then use that address in the call to the client

Submitted by MarkoD on Tue, 2020/01/21 - 06:00

#11

Thanks! I'm running on a GL-MT300N-V2, so I need to upload the Linux client to it via SSH and then run it like you said?

What would be great if you could provide multiple things to do after a single event, for example (note that I've used ; as a separator):

onUnbind=power_cycle_port;/root/unbind.sh "$ADDRESS$" "$DEVPATH$" "$SURPRISE_UNBOUND$"

Submitted by Michael on Tue, 2020/01/21 - 10:19

#12

Ok that is a mipsel device, however im not sure it has the space on the flash to install the mipsel client. Im not sure you will be able to do this with the cloudhub...

Submitted by MarkoD on Wed, 2020/01/22 - 06:29

#13

If I just put two separate entries in the server configuration file, would it work?

onUnbind=power_cycle_port
onUnbind=/root/unbind.sh "$ADDRESS$" "$DEVPATH$" "$SURPRISE_UNBOUND$"

Submitted by Michael on Wed, 2020/01/22 - 10:03

#14

No unfortunately that wont work, there is only one line possible. How about you just power_cycle_port on all unbinds? e.g onUnbind=power_cycle_port

Submitted by MarkoD on Thu, 2020/01/23 - 00:48

#15

I'd rather power cycle the USB only in case of a surprise unbound. However if I need to choose only one option, I'll just send the email and do a power reset manually in case the device gets unresponsive after the event, which is not always the case.

Submitted by MarkoD on Sat, 2020/01/25 - 05:41

#16

Michael, what is the easiest way of sending emails from the CloudHub firmware on a GL-MT300N-V2? Is there any suitable package built-in or do I have to install one?

Submitted by MarkoD on Sat, 2020/01/25 - 10:12

#18

Thanks! I guess I'll have to reinstall the package and reupload the script each time I flash a newer firmware, right?

Submitted by Michael on Sat, 2020/01/25 - 10:43

#19

I think it will remember what packages are installed via opkg.

Firmware upgrade also will keep /root/config.ini and /root/server.pem (if there) but other files will be lost so you will probably need to backup them beforehand

Submitted by MarkoD on Thu, 2020/01/30 - 07:58

#21

I'm trying out a USB hub (namely this one) powered by a 5V 2A adapter and I've noticed that when I try to power-cycle the USB port on the GL-MT300N-V2 (using power_cycle_port), the USB device does not disconnect (according to the system logs). Is it because the USB hub is self-powered and thus resetting the USB port practically does not do anything?

The SURPRISE_UNBOUND disconnection happened once a week, so I'll need to test this a couple of weeks in order to see if the powered-hub sloved the issue.

Submitted by Michael on Thu, 2020/01/30 - 10:24

#22

If only it was that simple :)

Almost all manufacturers skip PCB traces when implementing USB Hubs to save money.

So only these hubs work with power cycling https://github.com/mvp/uhubctl

Note: the USB Port on the GL-MT300NV2 also correctly power-cycles if you only have one device plugged into it. Its not on the list but it does power cycle correctly through a custom mechanism which virtualhere server knows about.

If you plug one of those into the GL-MT300V2 , the hub ports will also power-cycle correctly

Submitted by MarkoD on Thu, 2020/02/06 - 00:53

#23

Hi Michael, thanks for the link, I'll get a compatible USB hub. Meanwhile, recently I found these warnings in the logs:

Wed Feb 5 14:38:06 2020 user.warn vhusbdchglmt300nv2[1090]: Error 22 discarding urb 0xd53d60 for device /sys//devices/platform/101c1000.ohci/usb2/2-1, Invalid argument (abort endpoint)
Wed Feb 5 14:38:06 2020 user.warn vhusbdchglmt300nv2[1090]: Error 22 discarding urb 0xda5b70 for device /sys//devices/platform/101c1000.ohci/usb2/2-1, Invalid argument (abort endpoint)
Wed Feb 5 14:38:06 2020 user.warn vhusbdchglmt300nv2[1090]: Error 22 discarding urb 0xd53d60 for device /sys//devices/platform/101c1000.ohci/usb2/2-1, Invalid argument (abort endpoint)
Wed Feb 5 14:38:06 2020 user.warn vhusbdchglmt300nv2[1090]: Error 22 discarding urb 0xda5b70 for device /sys//devices/platform/101c1000.ohci/usb2/2-1, Invalid argument (abort endpoint)

Note that the USB device was connected directly to the GL unit at that time. Unlike SURPRISE UNBOUND, these warnings didn't leave the device unresponsive and communication worked okay afterwards (no port power-cycle was necessary).

Submitted by MarkoD on Thu, 2020/02/06 - 01:04

#24

Follow-up: The device didn't work okay after the warnings :-(. Should the powered USB hub solve both issues ( disabled by hub (EMI?), re-enabling... and also Error 22 discarding urb), or are they not related at all?

If the hub won't work, maybe an electro-magnetic interference is causing this or a hardware issue? We have 4 units like this one in production, and only one is malfunctioning.

Submitted by Michael on Thu, 2020/02/06 - 08:13

#25

Yeah its a $20 device dont waste time if its the only one broken just buy another one. I assumed you already did that, and it still wasnt working

Submitted by MarkoD on Thu, 2020/02/06 - 10:18

#26

Ok, I'll try a spare unit and will report back. On a side note, I tried to manually turn the GL-MT300N-V2's USB port off and on again via commands CUSTOM_EVENT,...,port=off and then port=on, however the port never turns back on again. The server log does not contain anything and the client's system messages only have one entry ("Change port power returned 0") when turning the port off.

Note that power_cycle_port works okay, however I would like to have control of the timeout, since I think the default 500 ms is too short for the USB device I am using.

Thanks again for your help and assistance, Mihael!

Submitted by Michael on Thu, 2020/02/06 - 14:05

#27

The timeout is 1 second between power off/on

In an event handler you can only use power_cycle_port

Submitted by MarkoD on Sun, 2020/06/14 - 03:37

#28

Hi Michael,

just letting you know that connecting a self-powered USB hub fixed the issues, so your suggestion was correct ;-). I've bought the HU3641V1 so I can restart the ports if anything happens, but I've been running 3 months without any issues.

Thanks again,

Marko

Submitted by Michael on Sun, 2020/06/14 - 09:58

#29

OK thats great, thanks for letting me know