Android ADB devices not connecting

I'm setting up a few USBIP servers on a raspberry pi, to share with a linux client on the network. On the rPi I am putting a bunch of Android devices that I will use the linux client to talk to via adb.

Unfortunately I'm running into a few problems, the main of which is that actually getting adb to talk to the devices is exceedingly flaky (last week it was working, today I can't get it to really work at all).

Here's what I'm seeing:

1) Rpi connects to the devices fine, and can generally share them out.
2) Running -t "LIST" on the client shows that the devices are shared out, and that AUTO USE is turned on (except when it decides to revert back and turn itself off, which is another issue).
3) Running lsusb shows the devices (again this is generally the case, occasionally I'll see the devices in the LIST command, but lsusb shows nothing).
4) Running adb devices shows 1 device connected fine, but the rest are set for ??????????? No Permissions. One would think this is just the Android device waiting for the permissions pop-up to be accepted, but the pop-up never comes. Additionally, the waiting pop-up generally will show the devices serial number followed by unauthorized, not ?????????? No Permissions.

This only occurs when running via this method. If I plug the devices in directly, it works 100% of the time as expected.

Anyone else see something similar, or able to direct me to what might need to be done?