• httpSess coredump just appeared

    From Nigel Reed@1:103/705 to GitLab issue in main/sbbs on Thu Jan 19 22:56:38 2023
    open https://gitlab.synchro.net/main/sbbs/-/issues/495

    <code>$ gdb /sbbs/exec/sbbs '/tmp/core.sbbs!httpSess.3560723'[Thread debugging using libthread_db enabled]Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".Core was generated by `/sbbs/exec/sbbs d'.Program terminated with signal SIGSEGV, Segmentation fault.#0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120120 ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.[Current thread is 1 (Thread 0x7f7ccadfa700 (LWP 3676796))](gdb) bt#0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120#1 0x00007f7d8f1fdd15 in __vfprintf_internal (s=s@entry=0x7f7ccadf3e80, format=format@entry=0x7f7d8fb9a5f1 "%ld\t%s\t%s\t%s\t%s\t%u\t%lu", ap=ap@entry=0x7f7ccadf3ff0, mode_flags=mode_flags@entry=0) at vfprintf-internal.c:1688#2 0x00007f7d8f210bca in __vasprintf_internal (result_ptr=0x7f7ccadf3fe0, format=0x7f7d8fb9a5f1 "%ld\t%s\t%s\t%s\t%s\t%u\t%lu", args=0x7f7ccadf3ff0, mode_flags=0) at vasprintf.c:57#3 0x00007f7d8fb6c993 in strListAppendFormat (list=0x7f7ccadf4110, format=0x7f7d8fb9a5f1 "%ld\t%s\t%s\t%s\t%s\t%u\t%lu") at str_list.c:321#4 0x00007f7d8f6a66f5 in mqtt_client_on (mqtt=0x7f7d8f43b440 <mqtt>, on=0, sock=141, client=0x0, update=0) at mqtt.c:628#5 0x00007f7d8f40a356 in client_off (sock=141) at websrvr.c:772#6 0x00007f7d8f4203be in http_session_thread (arg=0x0) at websrvr.c:6776#7 0x00007f7d8f37f609 in start_thread (arg=<optimized out>) at pthread_create.c:477#8 0x00007f7d8f2a4133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95(gdb) </code>
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Fri Jan 20 07:29:23 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3082

    Interesting.. this looks like the same crash that took down vert.synchro.net (on Windows) a couple of weeks ago. Good to know it's not a Windows-only issue!
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Nigel Reed@1:103/705 to GitLab note in main/sbbs on Fri Jan 20 08:23:37 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3083

    Of course, I just realized I recompiled sbbs on 17th Jan and the crash was on 16th so the core and binary files are not going to line up I expect so not sure if that dump is completely useful or not.I have another core from Jan 13th and then one from core.sbbs!termNode.3347010 on 15th. Let me know if you want a bt from those as new issues or I can add them here.
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Fri Jan 20 20:49:36 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3085

    BT's from those other core dumps could be still useful.
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Sat Jan 21 11:05:23 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3091

    So here's the BT from a crash on Windows today (using code from a week ago, so mqtt.c line numbers don't match current SHA), but it's clearly the same crash in strlen() by way of vasprintf():```> sbbs.dll!common_strnlen_c<unsigned char>(const unsigned char * const string, const unsigned int maximum_count) Line 36 C++ sbbs.dll!common_strnlen_simd<0,unsigned char>(const unsigned char * const string, const unsigned int maximum_count) Line 94 C++ sbbs.dll!common_strnlen<unsigned char>(const unsigned char * const string, const unsigned int maximum_count) Line 153 C++ sbbs.dll!strnlen(const char * string, unsigned int maximum_count) Line 165 C++ sbbs.dll!__crt_stdio_output::output_processor<char,__crt_stdio_output::string_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::string_output_adapter<char>>>::type_case_s_compute_narrow_string_length(const int maximum_length, char __formal) Line 2268 C++ sbbs.dll!__crt_stdio_output::output_processor<char,__crt_stdio_output::string_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::string_output_adapter<char>>>::type_case_s() Line 2255 C++ sbbs.dll!__crt_stdio_output::output_processor<char,__crt_stdio_output::string_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::string_output_adapter<char>>>::state_case_type() Line 1999 C++ sbbs.dll!__crt_stdio_output::output_processor<char,__crt_stdio_output::string_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::string_output_adapter<char>>>::process() Line 1644 C++ sbbs.dll!common_vsprintf<__crt_stdio_output::standard_base,char>(const unsigned __int64 options, char * const buffer, const unsigned int buffer_count, const char * const format, __crt_locale_pointers * const locale, char * const arglist) Line 163 C++ sbbs.dll!__stdio_common_vsprintf(unsigned __int64 options, char * buffer, unsigned int buffer_count, const char * format, __crt_locale_pointers * locale, char * arglist) Line 235 C++ [External Code] sbbs.dll!vasprintf(char * * strptr, const char * format, char * va) Line 60 C sbbs.dll!strListAppendFormat(char * * * list, const char * format, ...) Line 321 C sbbs.dll!mqtt_client_on(mqtt * mqtt, int on, int sock, client_t * client, int update) Line 614 C websrvr.dll!client_off(unsigned int sock) Line 772 C websrvr.dll!http_session_thread(void * arg) Line 6775 C websrvr.dll!invoke_thread_procedure(void(*)(void *) procedure, void * const context) Line 82 C++ websrvr.dll!thread_start<void (__cdecl*)(void *)>(void * const parameter) Line 115 C++ [External Code] mqtt->client_list = {first=0x0760c020 {data=0x06cb0710 next=0x077ca988 {data=0x06cd4f48 next=0x076666c0 {data=0x06cd49f0 ...} ...} ...} ...}client_list.mutex = {DebugInfo=0x00782620 {Type=0 CreatorBackTraceIndex=0 CriticalSection=websrvr.dll!0x01a945dc {DebugInfo=...} ...} ...}client_list.count = 9client_list.sem = 0x00000000list = 0x131e8af0 {0x07bf2cc0 "2152\tHTTP\tGuest\t98.7.221.143\t<no name>\t56584\t1674288933"}```I don't see anything obviously wrong and this problem only happens like once in every millions calls to mqtt_client_on(), so I'm probably just going to punt and use an alternative approach to appending a new formatted string to the (temporary) client_list.
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Mon Jan 23 12:11:51 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3132

    This crash is still occurring with the latest changes and I think I know why:the client_list maintained in the mqtt object has pointers to the username for each connected client and in the web server, that points to char buffer in an ephemeral http_session_t instance which is likely being freed already at this point in the code in same race-conditions. Changing the client_t definition to use a char array for the username rather than a pointer should fix this.
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Mon Jan 23 17:05:16 2023
    close https://gitlab.synchro.net/main/sbbs/-/issues/495
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)