Kinect Support

Submitted by martinisoft on #1

Dear VirtualHere Team member,

I am considering to introduce Microsoft Kinect devices to my home.
They would be used by my Home Automation system running on a Windows server to detect if someone is in the room, and VR.
My server is running in the basement and I have Cat5e leads to every room.
I cannot seem to find a USB Server that supports the Microsoft Kinect and I was wondering if this would work with VirtualHere.
When browsing trough this forum I concluded that running VirtualHere on a Raspberry PI will proberly not cut it, since the NIC and USB hub share the same bus.
I am willing to buy a Beaglebone for this, but I don't want to buy hardware (Beaglebone and Kinect) and Software (Virtualhere) and not having a working situation at the end.
Is there any information on if Kinect wil work trough Virtualhere?
If not, how high are the chances that it will be working after working with VirtualHere support when I have the hardware to test/troubleshoot?

thanks in advance for your response,

Martijn.
The Netherlands

Submitted by Michael on Thu, 2014/02/27 - 07:23

#2

From experience, it seems that the Kinect is sensitive to latency when it processes the image signals received via USB. I dont think the beagle will be adequate for this, i think it would need a 1gbps ethernet link and a fast multicore arm processor. The technology is not quite there yet in terms of speed - but lately i have seen boards like the cubietruck which may provide closer to the performance required...

Submitted by SBobowski on Wed, 2016/11/09 - 09:00

#3

Now that a few years have passed and the Beagle has new models. (and other options are available)
what are your thoughts on supporting the Kinect remotely?

Submitted by Michael on Wed, 2016/11/09 - 09:53

#4

I notice some cheap embedded boards like the orange pi 2 etc or pine64+ have at least quad core and gigabit ethernet and i think that would at least be a minimum hardware requirement, which rules out the rasbpberry and the beagle. However almost no cheap embedded boards have usb 3, except for odroid xu4 for about $70 dollars but i heard the kernel support is pretty bad.

If you can find a board with all those features and recent kernel support it might be worth a try.

Edit: there is a beagle x15 out now which has dual core 1.5ghz and usb 3 with good recent kernel support, but its really expensive and im not sure it would work even with that board but you could try it anyway if you are not bothered by the price. And ill give you a free license key so you can use the optmized version as i would like to know if the kinect works with this board if you go down this path..

Submitted by veelstekel on Thu, 2017/03/09 - 22:16

#5

At home; I have the:
* Rasperry pi Zero, Zero-W, 1, 2 and 3
* Cubox i4pro (1gbs ethernet / USB2.0)
* Fanless PC; based on Asus C60M1-I (1gbs ethernet / USB 2.0)
These could act as the USB server.

Last week I bought from HumbleBundle the Xbox 360 kinect magazines; and I was planning on having the Pi as my Kinect Server.
From the posts above I gather it will not work; but now I am interested if the i4pro of C60 would do the trick. (And still would validate that the Pi would not be able...)

Btw, as the Xbox 360 only supports USB 2.0; I don't expect that the server would require USB 3.0 connections.
(Perhaps it is different with the Xbox One?)

Anyway if a license is provided (perhaps capped with the vendor and product id's for the Xbox 360 Kinect for the platforms mentioned) I would be happy to test.
No commitment on timelines however ;)

Submitted by Michael on Fri, 2017/03/10 - 09:02

#6

I think your best bet would be the cubox i4pro with the latest kernel. The others are too weak to be useful. I can send you a test license if you email mail [at] virtualhere.com (mail[at]virtualhere[dot]com)

Submitted by veelstekel on Fri, 2017/03/10 - 18:06

#7

I have send a request for a test license; in case it fails I also have the following devices which we can test:
* Android Nvidia Shield Tablet (One of the more powerful Android devices available)
* Desktop with: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz; 6GB; USB2.0 running CentOS-7
* Supermicro X8SIL server; (running Windows) - however only connected to 100Mbs (Due to inhouse-cabling issues)
I expect this will not suffice as I read on the microsoft site the kinect will take about 50% of the USB bandwidth (thus approx 240Mbs)

For the client I am planning to use an i7 / 16Gbs laptop, which should suffice.

I read on the internet some people had issues with two Kinects on the same USB hub due to Bandwidth issues; however by lowering the resolution etc they were able to get them connected. I might try the same tricks as well.

Submitted by veelstekel on Fri, 2017/03/10 - 18:41

#8

This topic starts to get me thinking about compression; would it make sense to have a feature in vhusb to support compression (ie like zip to support lower bandwitdth). Or would this have a too significant overhead on the latency.

Submitted by Michael on Sun, 2017/03/12 - 20:58

#9

OK i sent you an email...

regarding compression, yes virtualhere uses lz4 as its really fast and works well... the stream is always sent/received in compressed format if the packet size is > 384 bytes (less than that and its not worth it)

Submitted by veelstekel on Mon, 2017/03/13 - 01:33

#10

Ok, I first baselined by having the Kinect connected directly to the laptop.
I used the SDK 1.7 as this seems to be the correct version for the Xbox 360 version of the Kinect.
Using the Kinect for Windows toolkit I was successful able to get a picture from the Kinect.

After that I installed the Virtualhere Client / Server with the license.

After connecting to the Hub I got a "Operation Not Permitted (-1)" error trying to use this device.
And another VirtualHere client Window with "There was a error using this device" followed by a "Error No Such Device (-19) trying to use this device"

Device Manager shows:
- Problem with Kinect for Windows Audio Control
- Working Kinect for Windows Camera (twice?)
- Working Kinect for Windows Device

Virtual Here Client Shows:
- Xbox Kinect Audio (c) Microsoft (In use by you)
- Xbox NUI Camera (In use by you)
- Xbox NUI Motor (In use by you)

Windows Kinect Explorer does not detect the sensor.
Any ideas for troubleshooting?

Submitted by veelstekel on Mon, 2017/03/13 - 01:54

#11

I removed a module which seemed to 'claim' the kinect on my Archlinux box (gspc_kinect or something).
After that I replugged the kinect, restarted virtualhere server.

Still the same issues however; this is from the log:
Sun Mar 12 16:50:38 2017 LOG_INFO >>> Starting VirtualHere USB Server v3.2.8 (Built: Mar 7 2017, 21:25:30)<<<
Sun Mar 12 16:50:38 2017 LOG_INFO Using configuration /root/vhusb/config.ini
Sun Mar 12 16:50:38 2017 LOG_INFO Server licensed to=d063b400601f max_devices=unlimited
Sun Mar 12 16:50:38 2017 LOG_INFO Using large URB's
Sun Mar 12 16:50:38 2017 LOG_DEBUG TCPServer starting...
Sun Mar 12 16:50:38 2017 LOG_INFO Listening on all network interfaces at port 7575
Sun Mar 12 16:50:38 2017 LOG_INFO Found High speed device [045e:02ad] "Microsoft, Xbox Kinect Audio, © 2011 Microsoft Corporation. All rights re" at address 111
Sun Mar 12 16:50:38 2017 LOG_INFO Found Full speed device [045e:02b0] "Microsoft, Xbox NUI Motor" at address 112
Sun Mar 12 16:50:38 2017 LOG_INFO Found High speed device [045e:02ae] "Microsoft, Xbox NUI Camera" at address 113
Sun Mar 12 16:50:39 2017 LOG_INFO Callback: VirtualHere USB Sharing_vhusb_tcplocal Name Registered
Sun Mar 12 16:50:58 2017 LOG_DEBUG 192.168.2.104 connected
Sun Mar 12 16:50:58 2017 LOG_INFO Device 111 [045e:02ad] BOUND to connection 1
Sun Mar 12 16:50:58 2017 LOG_INFO Device 112 [045e:02b0] BOUND to connection 1
Sun Mar 12 16:50:58 2017 LOG_ERR Error -1 resetting device 0x045e/0x02ae for capture
Sun Mar 12 16:50:58 2017 LOG_INFO Device 111 [045e:02ad] SURPRISE UNBOUND from connection 1
Sun Mar 12 16:50:58 2017 LOG_ERR Error binding device 113 [045e:02ae] to connection 1, BIND_ERROR
Sun Mar 12 16:50:58 2017 LOG_INFO Unmanaging device 111 [045e:02ad]
Sun Mar 12 16:50:59 2017 LOG_INFO Unmanaging device 113 [045e:02ae]
Sun Mar 12 16:51:00 2017 LOG_INFO Found High speed device [045e:02ad] "Microsoft, Xbox Kinect Audio, © 2011 Microsoft Corporation. All rights re" at address 111
Sun Mar 12 16:51:00 2017 LOG_INFO Device 111 [045e:02ad] BOUND to connection 1
Sun Mar 12 16:51:05 2017 LOG_INFO Found High speed device [045e:02ae] "Microsoft, Xbox NUI Camera" at address 113
Sun Mar 12 16:51:05 2017 LOG_ERR Error -19 resetting device 0x045e/0x02ae for capture
Sun Mar 12 16:51:05 2017 LOG_ERR Error binding device 113 [0000:0000] to connection 1, BIND_ERROR
Sun Mar 12 16:51:06 2017 LOG_INFO Device 113 [045e:02ae] BOUND to connection 1
Sun Mar 12 16:51:06 2017 LOG_INFO Connection 1 remotely disconnected abruptly (rx msg size)
Sun Mar 12 16:51:16 2017 LOG_INFO Device 112 [045e:02b0] UNBOUND from connection 1
Sun Mar 12 16:51:16 2017 LOG_INFO Device 111 [045e:02ad] UNBOUND from connection 1
Sun Mar 12 16:51:16 2017 LOG_INFO Device 113 [045e:02ae] UNBOUND from connection 1
Sun Mar 12 16:51:17 2017 LOG_INFO Connection 1 successfully removed (reason:timeout)
Sun Mar 12 16:51:38 2017 LOG_INFO Unmanaging device 111 [045e:02ad]
Sun Mar 12 16:51:39 2017 LOG_INFO Unmanaging device 112 [045e:02b0]
Sun Mar 12 16:51:39 2017 LOG_INFO Unmanaging device 113 [045e:02ae]
Sun Mar 12 16:51:42 2017 LOG_INFO Found Full speed device [045e:02b0] "Microsoft, Xbox NUI Motor" at address 112
Sun Mar 12 16:51:44 2017 LOG_INFO Found High speed device [045e:02ad] "Microsoft, Xbox Kinect Audio, © 2011 Microsoft Corporation. All rights re" at address 111
Sun Mar 12 16:51:45 2017 LOG_INFO Found High speed device [045e:02ae] "Microsoft, Xbox NUI Camera" at address 113
Sun Mar 12 16:52:13 2017 LOG_DEBUG 192.168.2.104 connected
Sun Mar 12 16:52:13 2017 LOG_INFO Device 112 [045e:02b0] BOUND to connection 6
Sun Mar 12 16:52:13 2017 LOG_ERR Error -1 resetting device 0x045e/0x02ad for capture
Sun Mar 12 16:52:13 2017 LOG_ERR Error binding device 111 [045e:02ad] to connection 6, BIND_ERROR
Sun Mar 12 16:52:13 2017 LOG_INFO Unmanaging device 111 [045e:02ad]
Sun Mar 12 16:52:13 2017 LOG_ERR Error -1 resetting device 0x045e/0x02ae for capture
Sun Mar 12 16:52:13 2017 LOG_ERR Error binding device 113 [045e:02ae] to connection 6, BIND_ERROR
Sun Mar 12 16:52:13 2017 LOG_INFO Unmanaging device 113 [045e:02ae]
Sun Mar 12 16:52:14 2017 LOG_INFO Found High speed device [045e:02ad] "Microsoft, Xbox Kinect Audio, © 2011 Microsoft Corporation. All rights re" at address 111
Sun Mar 12 16:52:15 2017 LOG_INFO Device 111 [045e:02ad] BOUND to connection 6
Sun Mar 12 16:52:20 2017 LOG_ERR Error -19 resetting device 0x0000/0x0000 for capture
Sun Mar 12 16:52:20 2017 LOG_ERR Error binding device 113 [0000:0000] to connection 6, BIND_ERROR
Sun Mar 12 16:52:20 2017 LOG_INFO Found High speed device [045e:02ae] "Microsoft, Xbox NUI Camera" at address 113
Sun Mar 12 16:52:20 2017 LOG_INFO Device 113 [045e:02ae] BOUND to connection 6

Submitted by Michael on Mon, 2017/03/13 - 10:17

#13

OK so its working now? Is the performance OK?

Submitted by veelstekel on Mon, 2017/03/13 - 17:50

#14

No, from the link above it seems possible to have a Kinect working with a Raspberry PI;
however I cannot get it to work with VirtualHere (See errors above). Let me know in case you want me to try something else.
(I could try another Clean CentOS-7 server).

Anything we can do with the BIND_ERRORS from the Log?

Submitted by Michael on Mon, 2017/03/13 - 18:25

#15

Yes to avoid the BIND_ERROR, right click on "Microsoft, Xbox NUI Motor" and select 'Custom Event handler..." then enter

onReset.$VENDOR_ID$.$PRODUCT_ID$=

then press OK and repeat that for each of the Microsoft,... devices listed then try again to use them. This will disable the USB reset command. The kinect devices dont seem to like this command.

Submitted by veelstekel on Sun, 2017/03/26 - 22:38

#16

Ok, with the provided tip the error is gone, however the sympoms are still the same.
I.e. Non working Kinect for Windows Audio Control, and two which do appear.
Kinect Toolkit still cannot find the kinect however.

I read somewhere on the internet that other USB servers are facing the same issues as the entire Kinect USB Hub would need to be exposed; is this possible?

Logfile:
[root@alarm vhusb]# cat vh.log
Sun Mar 26 14:36:55 2017 LOG_INFO >>> Starting VirtualHere USB Server v3.2.8 (Built: Mar 7 2017, 21:25:30)<<<
Sun Mar 26 14:36:55 2017 LOG_INFO Using configuration /root/vhusb/config.ini
Sun Mar 26 14:36:55 2017 LOG_INFO Server licensed to=d063b400601f max_devices=unlimited
Sun Mar 26 14:36:55 2017 LOG_INFO Using large URB's
Sun Mar 26 14:36:55 2017 LOG_DEBUG TCPServer starting...
Sun Mar 26 14:36:55 2017 LOG_INFO Listening on all network interfaces at port 7575
Sun Mar 26 14:36:55 2017 LOG_INFO Found Low speed device [03f0:0324] "LiteON, HP Basic USB Keyboard" at address 21
Sun Mar 26 14:36:55 2017 LOG_INFO Found High speed device [045e:02ad] "Microsoft, Xbox Kinect Audio, © 2011 Microsoft Corporation. All rights re" at address 111
Sun Mar 26 14:36:55 2017 LOG_INFO Found Full speed device [045e:02b0] "Microsoft, Xbox NUI Motor" at address 112
Sun Mar 26 14:36:55 2017 LOG_INFO Found High speed device [045e:02ae] "Microsoft, Xbox NUI Camera" at address 113
Sun Mar 26 14:36:56 2017 LOG_INFO Callback: VirtualHere USB Sharing_vhusb_tcplocal Name Registered
Sun Mar 26 14:37:15 2017 LOG_DEBUG 192.168.2.101 connected
Sun Mar 26 14:37:16 2017 LOG_INFO Device 21 [03f0:0324] BOUND to connection 1
Sun Mar 26 14:37:16 2017 LOG_INFO Executed "" for onReset.045e.02ad
Sun Mar 26 14:37:16 2017 LOG_INFO Device 111 [045e:02ad] BOUND to connection 1
Sun Mar 26 14:37:16 2017 LOG_INFO Executed "" for onReset.045e.02b0
Sun Mar 26 14:37:16 2017 LOG_INFO Device 112 [045e:02b0] BOUND to connection 1
Sun Mar 26 14:37:16 2017 LOG_INFO Executed "" for onReset.045e.02ae
Sun Mar 26 14:37:16 2017 LOG_INFO Device 113 [045e:02ae] BOUND to connection 1
Sun Mar 26 14:37:53 2017 LOG_INFO Device 21 [03f0:0324] UNBOUND from connection 1
Sun Mar 26 14:37:53 2017 LOG_INFO Device 111 [045e:02ad] UNBOUND from connection 1
Sun Mar 26 14:37:53 2017 LOG_INFO Device 112 [045e:02b0] UNBOUND from connection 1
Sun Mar 26 14:37:53 2017 LOG_INFO Device 113 [045e:02ae] UNBOUND from connection 1
Sun Mar 26 14:37:53 2017 LOG_INFO Device 21 [03f0:0324] BOUND to connection 1
Sun Mar 26 14:37:53 2017 LOG_INFO Executed "" for onReset.045e.02ad
Sun Mar 26 14:37:53 2017 LOG_INFO Device 111 [045e:02ad] BOUND to connection 1
Sun Mar 26 14:37:53 2017 LOG_INFO Executed "" for onReset.045e.02b0
Sun Mar 26 14:37:53 2017 LOG_INFO Device 112 [045e:02b0] BOUND to connection 1
Sun Mar 26 14:37:53 2017 LOG_INFO Connection 1 successfully removed (reason:server shutdown)
Sun Mar 26 14:37:54 2017 LOG_INFO Device 21 [03f0:0324] SURPRISE UNBOUND from connection 1
Sun Mar 26 14:37:54 2017 LOG_INFO Unmanaging device 21 [03f0:0324]
Sun Mar 26 14:37:55 2017 LOG_INFO Device 111 [045e:02ad] SURPRISE UNBOUND from connection 1
Sun Mar 26 14:37:55 2017 LOG_INFO Unmanaging device 111 [045e:02ad]
Sun Mar 26 14:37:55 2017 LOG_INFO Device 112 [045e:02b0] SURPRISE UNBOUND from connection 1
Sun Mar 26 14:37:55 2017 LOG_INFO Unmanaging device 112 [045e:02b0]
Sun Mar 26 14:37:56 2017 LOG_INFO Unmanaging device 113 [045e:02ae]
Sun Mar 26 14:37:57 2017 LOG_INFO >>> Shutdown <<<

Submitted by Michael on Mon, 2017/03/27 - 11:32

#17

Actually remembering from a few years back, i think the kinect needs firmware uploaded to it when first connected to tell it what to do. That firmware is uploaded by the microsoft kinect driver automatically somehow. I think they tried to extract this in a native linux kinect driver. I think it may still not be compatible because of this firmware issue. I checked google and found this page https://github.com/OpenKinect/libfreenect . It talks about the firmware required about halfway down the page. Virtualhere doesnt set this firmware but the microsoft driver should when its passed through virtualhere. However you need to makesure linux doesnt set the device configuration to 1 before this firmware is uploaded

So you need to stop the server and edit the config.ini of the server and add the line

ClaimPorts=1

then save the config.ini file and start the server and unplug/replug the kinect then try to use it via virtualhere. Hopefully this might allow the firmware to be remotely uploaded via the built-in windows kinect driver as the server linux kernel wont be setting configuration 1 before the firmware is uploaded.

Submitted by blaineam on Wed, 2021/07/07 - 08:12

#18

I cannot get it to work. device manager showed it at first and now it crashes upon attaching the devices. I know it shows as a xbox NIC and a few 0x... devices and the 0x... device is failing to connect now. Im using macOS big sur on an intel Mac. could macOS be restricting it? I have a raspberry pi I could try it with I suppose. from what I understand the license is per server which means I may be out the $49 Im hoping to get it working but could use your help in doing so.

Submitted by Michael on Wed, 2021/07/07 - 09:38

#19

Kinect is not supported, it is too sensitive to latency. Is kinect even supported on mac? I dont think so