Client and Server on same machine

I have a situation where I want to run the virtual here server and client on the same raspberry pi.  That is, my devices are plugged into the USB ports on the Pi where the VH server is running and sometimes I want to access them from that machine and other times I want to access them remotely.  When I run the client on the server machine and claim one of the devices, the machine doesn't see the device.  Of course, if I don't run the client, the server sees the devices as natively plugged in.  I know this sounds obscure and I can confuse myself but hopefully you understand what I mean.  In general, I want to be able to access the devices through VH even though they are physically connected to the machine.  Thanks for your help.