Rpi3 vs Rpi4 usb latency?

Hello! I recently updated from a Rpi3 to an Rpi4 to take advantage of USB3, newer hardware etc. I have an interesting use case that VH works perfectly with - I have a home flight simulator but it's not exactly located close to my PC for typical USB connections. I am running a 30ft USB 3 extension cable which works, but instead of running multiple extension cables for what I need, VH works great to connect a hub back to my PC for peripherals. The issue I've run into is spikes in latency on the pi4 which didn't exist on the pi3. I mainly noticed my flight yoke started to disconnect on a test flight, when I checked the yokes software I noticed it was reporting high latency spikes intermittently so I performed some testing. On the USB extension cable reports around a 22ms average on polling spiking to 60, on the Pi3 - it was around 45 - spiking very rarely to 80 ms which is still more than fine. The Pi4 however seems to average 23-24ms however there are intermittent spikes to 2000, to 9000 ms on the polling. From the network perspective, my PC back to the pi4 (over wired ethernet, both on same gig switch) is only around 1.9ms with no spikes so this doesn't seem to be anything network related that I can tell.

All other peripherals seemed to work fine - even using an USB to HDMI adapter for 2 displays over VH. I could leave it on the wired extension and use VH for other devices which isn't a big deal - was mainly wondering if there's any troubleshooting worthwhile to do to maybe sort the cause. I've tried the 32bit and 64 versions of the rpi OS without any change.

Any ideas are appreciated - thanks!

-Chris