Client crash on particular microcontrollers on windows

Hello.
I have a problem that's sadly very hard to reproduce and leaves behind no log trace.

VH client often crashes on windows when connecting to MCX N23x and MCXA156 microcontrollers. There were other devices used with the same environment (different resberries, but same setup) and they don't produce the crash.

The VH server Is running on a Raspberry Pi 4.

The crash happens when trying to initiate the communication with the device on application level. The VH connection is established successfully, but crashes after data starts flowing over the serial port.

There is no mention in the client's log of any disconnect, from the server there doesn't seem to be anything of note.
The crash is especially pesky, because it has lasting effects. The exact behavior can differ a bit, but in majority cases it's as follows:

The connected device stays visible in device manager, and windows still seems to think that it is connected and cannot be removed, neither via GUI or directly through command line (pnputil).
After restarting VH, no device including the one which caused the crash can be connected to. VH will report them as acquired. but OS will not detect them.
This only gets fixed after a PC reset, which usually takes very long time, eventually leading into an OS crash. Most likely related to the "ghost" device left behind after the crash. Even OS can't handle it

Sometimes the device can be manually removed, in which case VH also continues to be usable. This is pretty rare, so I haven't been able to figure out any connected circumstances that would decide which of the two cases happens. I'm also not fully sure if the next info down bellow is the same for this case.

The crash does get logged in System Event messages. It's a fault when VH is using library C:\Windows\System32\ucrtbase.dll.
One entry is about the crash itself, one from Windows Error Reporting

Transcript of the crash report:
Faulting application name: vhui64.exe, version 5.8.2.0, time stamp 0x6783273d
Faulting module name ucrtbase.dll, version: 10.0.19041.3636, time stamp: 0x81cf5d89
Exception code: 0xc0000005
Fault offset: 0x0000000000026ed1
Faulting process id: 0xa08
Faulting application start time: 0x01db68e09e345551
Faulting application path: *VH client path*
Faulting module path: C:\Windows\System32\ucrtbase.dll
Faulting package full name:
Faulting package-relative application ID:

Transcript of the WER report containing mostly the same info:

Problem signature:
P1: vhui64.exe
P2: 5.8.2.0
P3: 6783273d
P4: ucrtbase.dll
P5: 10.0.19041.3636
P6: 81cf5d89
P7: c0000005
P8: 0x0000000000026ed1

Attached to the latter report is also this wer file, but nothing else (file should stay up for 7 days, i can send it again if it expires, or just send the contents, it's 200 lines):
https://tmpsend.com/hrccVij1

.

Try this:

Stop and exit the virtualhere client if it is running
Download and run https://github.com/jrfonseca/drmingw and install it as the crash handler
Download this debug version of the virtualhere client and run it https://www.virtualhere.com/sites/default/files/usbclient/test/vhui64.exe
When it crashes drmingw should popup a stacktrace. Post that here thanks.

Debug info

I finally managed to catch it. Sorry it took so long. Had to do it through attaching to process, not throug JIT but through attaching to process manually.
The problem might also be related to VH running through SSH, windows does run the program through some desktop-less mode, which might change things.

Anyway the whole thing is pretty long. The important parts seem to be this at the end:

vhui64.exe caused an Unknown [0xC0000374] Exception at location 00007FFAB891CA39 in module ntdll.dll.

AddrPC Params
00007FFAB891CA39 0000004641300000 00007FFAB8993908 00007FFAB8969C30 ntdll.dll!RtlReportFatalFailure+0x9
00007FFAB891CA03 0000000000000000 00007FFAB89938B0 0000000000000008 ntdll.dll!RtlReportCriticalFailure+0x97
00007FFAB8925A9A 0000000000000008 0000018681CCA640 0000018681280000 ntdll.dll!RtlpHeapHandleError+0x12
00007FFAB8925D7A 0000018681280000 0000018681CCA650 0000018682873900 ntdll.dll!RtlpHpHeapHandleError+0x7a
00007FFAB8931D75 00007FF7946E46E8 00000186823C06F0 0000018681CC9F40 ntdll.dll!RtlpLogHeapFailure+0x45
00007FFAB884C38C 0000018681CCA640 0000018681280000 00000046412FF8B0 ntdll.dll!RtlpFreeHeapInternal+0x84c
00007FFAB884AFD1 0000018682873900 00000186825DC560 00017FFAB5720000 ntdll.dll!RtlFreeHeap+0x51
00007FFAB62A364B 0000000000000000 0000000000000000 0000000000000000 ucrtbase.dll!_free_base+0x1b
00007FF792B5A76D 0000018682873900 00000186825DC560 0000000000000000 vhui64.exe!processWindowsHIDInterfaceArrived+0x1ed
00007FF792F11AAC 0000018682873900 0000000000000000 0000000000000000 vhui64.exe!pthread_create_wrapper+0x8c
00007FFAB62B9333 0000000000000000 0000000000000000 0000000000000000 ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>+0x93
00007FFAB655257D 0000000000000000 0000000000000000 0000000000000000 KERNEL32.DLL!BaseThreadInitThunk+0x1d
00007FFAB886AF08 0000000000000000 0000000000000000 0000000000000000 ntdll.dll!RtlUserThreadStart+0x28
vhui64.exe caused an Unknown [0xC0000374] Exception at location 00007FFAB891CA39 in module ntdll.dll.
AddrPC Params
00007FFAB891CA39 0000004641300000 00007FFAB8993908 00007FFAB8969C30 ntdll.dll!RtlReportFatalFailure+0x9
00007FFAB891CA03 0000000000000000 00007FFAB89938B0 0000000000000008 ntdll.dll!RtlReportCriticalFailure+0x97
00007FFAB8925A9A 0000000000000008 0000018681CCA640 0000018681280000 ntdll.dll!RtlpHeapHandleError+0x12
00007FFAB8925D7A 0000018681280000 0000018681CCA650 0000018682873900 ntdll.dll!RtlpHpHeapHandleError+0x7a
00007FFAB8931D75 00007FF7946E46E8 00000186823C06F0 0000018681CC9F40 ntdll.dll!RtlpLogHeapFailure+0x45
00007FFAB884C38C 0000018681CCA640 0000018681280000 00000046412FF8B0 ntdll.dll!RtlpFreeHeapInternal+0x84c
00007FFAB884AFD1 0000018682873900 00000186825DC560 00017FFAB5720000 ntdll.dll!RtlFreeHeap+0x51
00007FFAB62A364B 0000000000000000 0000000000000000 0000000000000000 ucrtbase.dll!_free_base+0x1b
00007FF792B5A76D 0000018682873900 00000186825DC560 0000000000000000 vhui64.exe!processWindowsHIDInterfaceArrived+0x1ed
00007FF792F11AAC 0000018682873900 0000000000000000 0000000000000000 vhui64.exe!pthread_create_wrapper+0x8c
00007FFAB62B9333 0000000000000000 0000000000000000 0000000000000000 ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>+0x93
00007FFAB655257D 0000000000000000 0000000000000000 0000000000000000 KERNEL32.DLL!BaseThreadInitThunk+0x1d
00007FFAB886AF08 0000000000000000 0000000000000000 0000000000000000 ntdll.dll!RtlUserThreadStart+0x28

But there also seem to be a lot (at least 40+) threads that seem stuck, although I might be misunderstanding the meaning here. All of them look very similar, with the same stack trace

AddrPC Params
00007FFAB88B3CC4 0000018682576690 0000000000000000 00000186824800C0 ntdll.dll!NtWaitForAlertByThreadId+0x14
00007FFAB887969B 00000186825DA240 0000000000000000 0000000000000000 ntdll.dll!RtlSleepConditionVariableSRW+0x13b
00007FFAB5F3F5E9 00000186824F5B60 0000018682576690 00000186824F5B50 KERNELBASE.dll!SleepConditionVariableSRW+0x29
00007FF792F3F713 0000000000000000 00007FFAB884C330 00000186824F5AE0 vhui64.exe!std::__1::__libcpp_condvar_wait+0x13
00007FF792F18D02 0000000000000000 0000000000000000 000001868257712F vhui64.exe!std::__1::condition_variable::wait+0x12
00007FF792ADB36D 0000000000000000 0000000000000000 00000186825DA138 vhui64.exe!Connection::msgProcessorThreadRoutine+0x5d
00007FF792ADE832 000000463A4FF7B8 00007FF792ADE593 00000186825DA120 vhui64.exe!std::__1::__invoke[abi:ne180100]<void (Connection::*)(), Connection*, , void>+0x72
00007FF792ADE7B3 000053BB05B2E8C7 00007FF792F3F8B2 0000000000000000 vhui64.exe!std::__1::__thread_execute[abi:ne180100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (Connection::*)(), Connection*, 2ull>+0x33
00007FF792ADE539 00000186825DA240 0000000000000000 0000000000000000 vhui64.exe!std::__1::__thread_proxy[abi:ne180100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (Connection::*)(), Connection*> >+0x69
00007FFAB62B9333 0000000000000000 0000000000000000 0000000000000000 ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>+0x93
00007FFAB655257D 0000000000000000 0000000000000000 0000000000000000 KERNEL32.DLL!BaseThreadInitThunk+0x1d
00007FFAB886AF08 0000000000000000 0000000000000000 0000000000000000 ntdll.dll!RtlUserThreadStart+0x28

dropping the full thing here
https://tmpsend.com/qgJSmDVG

.

Virtualhere requires GUI mode (it is not a console program) so that could be the issue.

"VH running through SSH" that is not really supported unless you setup sshd correctly. E.g https://stackoverflow.com/questions/59880794/starting-gui-programs-via-openssh-on-windows because it needs to be able to access the user session.

Virtualhere is expecting to be notified of HID devices arriving and relies on GUI Windows APIs to determine what it is.

(All those 40+ threads are running ok but paused because of the crash)

SSHD

Understood. Will set up sshd and report back.