I have noticed that, whilst connected via SteamLink with virtualhere for steam link, everytime I press (on a USB Nintendo 64 clone pad) down on the right yellow dpad, mouse starts to drift up.
If I disconnect and reconnect the issue disappears until I use the key again.
Also weirdly, Start button seems to send "esc" key.
.
It seems perhaps the dpad buttons are not configured correctly. I think the steam big picture client has the ability to map the controller buttons. Could you double check that mapping.