VRS server on RPI hangs, accepts no connections

[leemoore1966]

Hi All

I have a scenario where VRS is running happily on an RPI for a couple of days.
At some undefined time it will stop accepting incoming TCP connections.

I have no idea how to debug this or provide further feedback, once the process is running all I know is to run something like strace on the running process, unfortunately as this is running under an emulator (mono), I am not sure what help this is ?

The only solution right now is to restart - which is un-satisfactory, and requires constant attention by the maintainer

is there a recommendation for firstly adding some tracing/debug to isolate the underlying issue, at the moment its a bit of a black box and I have no way of providing further information to get to a solution

I should add this is version 2.4.4.36598 running mono 5.18.0.240
Also this is running on a headless machine using -nogui

Thx
Lee

[leemoore1966]

I have an update, and would appreciate any feedback
It appears that VRS would not startup, and was giving me the following output

WARNING: The runtime version supported by this application is unavailable.
Using default runtime: v4.0.30319
* Assertion at metadata.c:1048, condition `index < meta->heap_strings.size' not met

Stacktrace:

/proc/self/maps:
00010000-003e1000 r-xp 00000000 b3:07 272680 /usr/bin/mono-sgen
003f1000-003f5000 r--p 003d1000 b3:07 272680 /usr/bin/mono-sgen
003f5000-003f8000 rw-p 003d5000 b3:07 272680 /usr/bin/mono-sgen
003f8000-0047f000 rw-p 00000000 00:00 0
00c89000-00d7b000 rw-p 00000000 00:00 0 [heap]
73f10000-74015000 r--p 00000000 b3:07 525078 /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll
74015000-74117000 r--p 00000000 b3:07 528514 /home/pi/vrs/bin/VirtualRadar.WebSite.dll
74117000-7415a000 r--p 00000000 b3:07 528524 /home/pi/vrs/bin/VirtualRadar.Database.dll
7415a000-741cf000 r--p 00000000 b3:07 528973 /home/pi/vrs/bin/VirtualRadar.Library.dll
741cf000-74443000 r--p 00000000 b3:07 525014 /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
74443000-74707000 r--p 00000000 b3:07 403714 /usr/lib/mono/gac/System.Windows.Forms/4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
74707000-747d5000 r--p 00000000 b3:07 528512 /home/pi/vrs/bin/VirtualRadar.WinForms.dll
747d5000-74f84000 r-xp 00000000 b3:07 402413 /usr/lib/mono/aot-cache/arm/mscorlib.dll.so
74f84000-74f94000 ---p 007af000 b3:07 402413 /usr/lib/mono/aot-cache/arm/mscorlib.dll.so
74f94000-74f95000 rw-p 007af000 b3:07 402413 /usr/lib/mono/aot-cache/arm/mscorlib.dll.so
74f95000-74fa9000 rw-p 00000000 00:00 0
74fa9000-75400000 r--p 00000000 b3:07 404236 /usr/lib/mono/4.5/mscorlib.dll
75400000-75c00000 rw-p 00000000 00:00 0
75c00000-75c01000 ---p 00000000 00:00 0
75c01000-76400000 rwxp 00000000 00:00 0
76400000-76800000 rw-p 00000000 00:00 0
76805000-76806000 rw-p 00000000 00:00 0
76806000-76809000 r--p 00000000 b3:07 528510 /home/pi/vrs/bin/VirtualRadar.Interop.dll
76809000-76813000 r--p 00000000 b3:07 528513 /home/pi/vrs/bin/VirtualRadar.WebServer.dll
76813000-76889000 r--p 00000000 b3:07 524941 /usr/lib/mono/gac/System.Drawing/4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
Memory around native instruction pointer (0x76da9f70):
0x76da9f60 00 10 a0 e1 58 04 0c e5 43 7f a0 e3 00 00 00 ef ....X...C.......
0x76da9f70 01 0a 70 e3 00 30 a0 e1 08 00 00 8a 03 00 a0 e1 ..p..0..........
0x76da9f80 80 80 bd e8 00 00 50 e3 f6 ff ff ca 02 31 c0 e3 ......P......1..
0x76da9f90 00 00 53 e3 00 00 60 12 01 00 a0 01 f1 ff ff ea ..S...`.........

Native stacktrace:

Pkilling 0x76b1b450 from 0x76fee000
Entering thread summarizer pause from 0x76fee000
Finished thread summarizer pause from 0x76fee000.

Waiting for dumping threads to resume


Debug info from gdb:

[New LWP 21209]
[New LWP 21208]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
0x76ef9ef8 in __libc_waitpid (pid=21221, stat_loc=0x7eabc528, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40
40 ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
Id Target Id Frame
3 Thread 0x763ff450 (LWP 21208) "SGen worker" 0x76ef57a4 in __pthread_cond_wait (Cannot access memory at address 0x50
cond=0x403408, mutex=0x4033cc) at pthread_cond_wait.c:187
2 Thread 0x76b1b450 (LWP 21209) "Finalizer" 0x76ef7a40 in do_futex_wait (isem=isem@entry=0x3fb690) at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:48
* 1 Thread 0x76fee000 (LWP 21207) "mono" 0x76ef9ef8 in __libc_waitpid (pid=21221, stat_loc=0x7eabc528, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40

Thread 3 (Thread 0x763ff450 (LWP 21208)):
Cannot access memory at address 0x50
#0 0x76ef57a4 in __pthread_cond_wait (cond=0x403408, mutex=0x4033cc) at pthread_cond_wait.c:187
#1 0x002f7614 in ?? ()
Cannot access memory at address 0x50
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 0x76b1b450 (LWP 21209)):
#0 0x76ef7a40 in do_futex_wait (isem=isem@entry=0x3fb690) at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:48
#1 0x76ef7af4 in __new_sem_wait (Cannot access memory at address 0x0
sem=0x3fb690) at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:69
#2 0x002891e0 in ?? ()
Cannot access memory at address 0x0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0x76fee000 (LWP 21207)):
#0 0x76ef9ef8 in __libc_waitpid (pid=21221, stat_loc=0x7eabc528, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40
#1 0x001166e8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

./startvrs: line 89: 21207 Aborted mono bin/VirtualRadar.exe -nogui < /dev/null