• A47 System won't poll / Runtime error 216

    From Chris Hizny@1:105/420 to All on Tue Sep 28 09:11:52 2021
    This was a problem for me on A46. I had hoped an upgrade to A47 would make a difference and it doesn't appear to.

    2021.28.09 16:00:10 MUTIL 000 Runtime error 216

    What is this?

    The effect is the same when I stop and restart mis and the poll jobs (about every 5 minutes, one for each network) runs. The first poll will work okay, but then subsequently every poll fails:

    + 2021.09.28 16:00:07 1-Connecting to net1.fsxnet.nz on port 24556
    + 2021.09.28 16:00:07 1-Using address 219.89.83.33
    + 2021.09.28 16:00:08 1-Connected by IPV4 to 219.89.83.33
    + 2021.09.28 16:00:08 1-System Agency + Risa HUB
    + 2021.09.28 16:00:08 1-SysOp Paul Hayton
    + 2021.09.28 16:00:08 1-Location Dunedin, New Zealand
    + 2021.09.28 16:00:08 1-Mailer binkd/1.1a-112/Linux binkp/1.1
    + 2021.09.28 16:00:08 1-Receiving: 53397b12.we0 (849 bytes)
    + 2021.09.28 16:00:09 1-Session ended (0 sent, 1 rcvd, 0 skip)
    + 2021.09.28 16:00:10 Polled 1 systems

    --------------------- POLL v1.12 A47 2021/09/24 Tue, Sep 28 2021 (loglevel 1) + 2021.09.28 16:05:07 Poll BINKP node via address lookup: 21:1/10
    + 2021.09.28 16:05:09 Polled 0 systems

    (all subsequent polls will show "Polled 0 systems"). It doesn't appear to be one network. Depending on what part of the hour it is, it will always poll whatever network is scheduled correctly once, and then fail for all subsequent ones.

    I do not know if this is normal or not, but the other thing that happens is .bsy files appear in a whole bunch of my echomail/out folders:

    mystic@shibboleths:/mystic$ find . -name *.bsy ./echomail/out/fidonet.001/00da02bc.bsy ./echomail/out/agoranet.02e/0001009c.bsy
    ./echomail/out/tqwnet.539/00030064.bsy
    ./echomail/out/micronet.26a/012c0001.bsy ./echomail/out/spooknet.2bc/00640000.bsy
    ./echomail/out/primary/00010064.bsy
    ./echomail/out/primary/0001000a.bsy
    ./echomail/out/metronet.019/00190000.bsy ./echomail/out/weednet.1a4/00020001.bsy
    ./semaphore/mutil.bsy
    ./semaphore/mis.bsy

    Note also that mutil.bsy stays there even though mis is not running between polls; I assume this runtime error causes it to die before it can remove that file.

    The mutil.log file shows this:

    + Sep 28 16:10:25 Waiting for BUSY nodes
    + Sep 28 16:11:25 Results: Cannot import. Some nodes are BUSY in 60.06s
    ! Sep 28 16:11:25 Status: FATAL
    + Sep 28 16:11:25 Process: Importing EchoMail
    + Sep 28 16:11:25 Waiting for BUSY nodes
    + Sep 28 16:12:25 Results: Cannot import. Some nodes are BUSY in 60.06s
    ! Sep 28 16:12:25 Status: FATAL
    + Sep 28 16:12:25 Process: Merging Nodelists
    + Sep 28 16:12:25 Results: Merged 0 of 0 nodelist(s) in 0.00s
    + Sep 28 16:12:25 Shutdown Normal (0)


    Currently dead in the water and unsure how to proceed.

    --- Mystic BBS v1.12 A47 2021/08/29 (Raspberry Pi/32)
    * Origin: 2o fOr beeRS bbS>>20ForBeers.com:1337 (1:105/420)
  • From Paul Hayton@3:770/100 to Chris Hizny on Wed Sep 29 11:44:17 2021
    On 28 Sep 2021 at 09:11a, Chris Hizny pondered and said...

    I do not know if this is normal or not, but the other thing that happens is .bsy files appear in a whole bunch of my echomail/out folders:

    If you run 'mis poll killbusy' but close MIS down first before you do

    Then restart and see how you go.

    Best, Paul

    ... A black cat crossing your path signifies that the animal is going somewhere

    --- Mystic BBS v1.12 A47 2021/09/24 (Linux/64)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (3:770/100)
  • From Chris Hizny@1:218/860 to Paul Hayton on Wed Sep 29 08:24:30 2021
    If you run 'mis poll killbusy' but close MIS down first before you do Then restart and see how you go.

    Thanks! It turned out the central issue was my system rebooted in the middle of a poll, leaving corrupt stuff in my echomail folders (determined this by trying to unzip everything and a bunch of things would not unzip) which was causing Mystic to soil the proverbial bed. As it crashed, it was leaving those .bsy files behind. I cleaned out my echomail folders and now it seems to be humming along. I am describing this here just in case someone else searches on the problem in the future.

    I will remember the killbusy method in the future; I was using a for loop to find the files and rm them previously.

    --- Mystic BBS v1.12 A47 2021/09/24 (Linux/64)
    * Origin: Shipwrecks & Shibboleths [San Francisco, CA - USA] (1:218/860)
  • From g00r00@1:129/215 to Chris Hizny on Wed Sep 29 15:27:25 2021
    Thanks! It turned out the central issue was my system rebooted in the middle of a poll, leaving corrupt stuff in my echomail folders
    (determined this by trying to unzip everything and a bunch of things
    would not unzip) which was causing Mystic to soil the proverbial bed.

    It sounds like you may have had some sort of corrupted packet of some type that Mystic couldn't gracefully "fail" with. Do you still have the file MUTIL was crashing on by chance or any of the files that were clogged up in your folder? I would like to see if I could use it to reproduce the crash and handle it more gracefully but I probably wouldn't have anything to go on without the actual file that was causing MUTIL to crash.

    I will remember the killbusy method in the future; I was using a for
    loop to find the files and rm them previously.

    Yeah this function exists for exactly this purpose. If Mystic gets shutdown in the middle of doing stuff, sometimes things can get stuck in a weird state. The killbusy command will clear all that mess up for you to and try to give you a clean slate, but it should really only be needed in those circumstances.
    Mystic is also supposed to "self-heal" after some period of time, but if a corrupted packet was causing MUTIL to crash over and over it would never been able to do that - the self healing of course would require it not shitting on itself before it can heal lol :)

    ... No honey, I can't eat with the family. My computer gets lonely!

    --- Mystic BBS v1.12 A47 2021/09/23 (Windows/64)
    * Origin: Sector 7 * Mystic WHQ (1:129/215)
  • From Chris Hizny@1:218/860 to g00r00 on Thu Sep 30 00:36:29 2021
    It sounds like you may have had some sort of corrupted packet of some
    type that Mystic couldn't gracefully "fail" with. Do you still have the file MUTIL was crashing on by chance or any of the files that were
    clogged up in your folder? I would like to see if I could use it to reproduce the crash and handle it more gracefully but I probably

    I had hoped one of my automated backups had copies when I went to sleep in the middle of the crisis, but apparently they don't. Next time I will save them. I didn't think much of the situation when it occurred since it was my fault.

    In any case, things are working fine now.

    Thanks for your response, and all of your work on Mystic all these years. I appreciate it.

    --- Mystic BBS v1.12 A47 2021/09/24 (Linux/64)
    * Origin: Shipwrecks & Shibboleths [San Francisco, CA - USA] (1:218/860)
  • From Chris Hizny@1:218/860 to g00r00 on Sun Oct 24 10:25:18 2021
    It sounds like you may have had some sort of corrupted packet of some
    type that Mystic couldn't gracefully "fail" with. Do you still have the file MUTIL was crashing on by chance or any of the files that were
    clogged up in your folder? I would like to see if I could use it to reproduce the crash and handle it more gracefully but I probably
    wouldn't have anything to go on without the actual file that was causing MUTIL to crash.

    Howdy. Well, it happened again, but this time I got granular and preserved the problem file. When I deleted it, all of the backlog processed fine and things are back to normal.

    Polling produces:

    Toss FDN/TIC Files 0 import, 0 toss, 0 hatch, 0 bad DONE 0.01s
    Importing EchoMail Importing 9FC80759.PKT (954:895/1 t WORKING

    Runtime error 216 at $0000000000443721
    $0000000000443721

    At this point there are .bsy files left behind for every message network and nothing works from this point on.

    ----------

    This was the problem file:

    Archive: 9DBACD4D.FR2
    inflating: ./unzip/9FC80759.PKT

    ----------

    I preserved the file for inspection:

    http://shibboleths.org/pub/9DBACD4D.FR2


    I am running on Ubuntu. Don't know if that matters.

    Distributor ID: Ubuntu
    Description: Ubuntu 20.04.3 LTS
    Release: 20.04
    Codename: focal

    --- Mystic BBS v1.12 A47 2021/09/24 (Linux/64)
    * Origin: Shipwrecks & Shibboleths [San Francisco, CA - USA] (1:218/860)
  • From Jay Harris@1:229/664 to g00r00 on Sun Oct 24 08:43:31 2021
    On 24 Oct 2021, Chris Hizny said the following...

    Howdy. Well, it happened again, but this time I got granular and preserved the problem file.

    Toss FDN/TIC Files 0 import, 0 toss, 0 hatch, 0 bad DONE 0.01s
    Importing EchoMail Importing 9FC80759.PKT (954:895/1 t WORKING

    I also got a bad packet from 954:895/1:

    ----------------- MUTIL v1.12 A47 2021/09/29 Tue, Oct 19 2021 (loglevel 2)
    + Oct 19 19:06:18 Startup using mailin.ini
    + Oct 19 19:06:18 Process: Toss FDN/TIC Files
    + Oct 19 19:06:18 Waiting for BUSY nodes
    + Oct 19 19:06:18 Scanning Hatches
    + Oct 19 19:06:18 Results: 0 import, 0 toss, 0 hatch, 0 bad in 0.01s
    + Oct 19 19:06:18 Process: Importing EchoMail
    + Oct 19 19:06:18 Waiting for BUSY nodes
    + Oct 19 19:06:19 Extracting 512A62CA.TU1
    + Oct 19 19:06:19 Importing 90F59CDD.PKT (954:895/1 to 954:895/27)
    + Oct 19 19:06:19 Shutdown Corrupted memory (216)

    Things started backing up after this as it said mutil was already running:

    + Oct 19 19:18:39 Results: Cannot import. Some nodes are BUSY in 60.09s
    + Oct 19 19:19:39 Results: Cannot import. Some nodes are BUSY in 60.10s
    + Oct 19 19:20:49 Results: Cannot import. Some nodes are BUSY in 60.08s
    + Oct 19 19:21:49 Results: Cannot import. Some nodes are BUSY in 60.09s

    Moving 512A62CA.TU1 out of the way and runing ./mis poll killbusy allowed the backlog to get processed.

    Here's a copy of 512A62CA.TU1:

    https://drive.google.com/file/d/1OUXGp6XSj9LZUUv9VhTrzfd9GW1aljUw/


    Jay

    ... Yield to temptation, it may not pass your way again. - L. Long

    --- Mystic BBS v1.12 A47 2021/09/29 (Raspberry Pi/32)
    * Origin: Northern Realms (1:229/664)
  • From g00r00@1:129/215 to Chris Hizny on Sun Oct 24 21:17:33 2021
    Howdy. Well, it happened again, but this time I got granular and preserved the problem file. When I deleted it, all of the backlog processed fine and things are back to normal.

    Awesome job catching this and remembering to pass it along thank you very much!

    I can't say what is causing the PKT file to be corrupted but Mystic doesn't read it correctly and neither could a whole bunch of other PKT viewers that I tried it with.

    I did reproduce the MUTIL crash though on my Windows system and I was able to put some code in place to make sure Mystic doesn't crash when processing that PKT now!

    This will be in the next A47 build.

    Thanks for your help tracking this one down!

    ... What hair color do they put on the driver's licenses of bald men?

    --- Mystic BBS v1.12 A47 2021/10/22 (Windows/32)
    * Origin: Sector 7 * Mystic WHQ (1:129/215)
  • From g00r00@1:129/215 to Jay Harris on Sun Oct 24 21:21:47 2021
    I also got a bad packet from 954:895/1:

    Thanks the fix I put in for the last packet also seems to fix the crash with this one too. I don't know what is going on with those PKT but they are not in a format Mystic is able to read.

    ... Running Windows is better than washing them!

    --- Mystic BBS v1.12 A47 2021/10/22 (Windows/32)
    * Origin: Sector 7 * Mystic WHQ (1:129/215)