• hpt long subjects and bad packets

    From Zhenja Kaliuta@2:4500/1.59 to All on Fri Jan 10 17:37:27 2020
    Hi!

    In case of subject longer then allowed 71 + NUL bytes HPT send the whole
    PKT with the incorrect message to bads.

    Would it be more practical to be more error tolerant and just cut it to
    the allowed limit, or it's considered as modification of messages and
    forbidden by some document?

    I'm wondering since fidogate has different logic and may be I'll have to
    fix it, even if I personally prefer to receive shorten subject than no
    message at all.
    --- Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
    * Origin: Somewhere in the North (2:4500/1.59)
  • From Michael Dukelsky@2:5020/1042 to Zhenja Kaliuta on Fri Jan 10 22:23:46 2020
    Hello Zhenja,

    Friday January 10 2020, Zhenja Kaliuta wrote to All:

    In case of subject longer then allowed 71 + NUL bytes HPT send the
    whole PKT with the incorrect message to bads.

    Would it be more practical to be more error tolerant and just cut it
    to the allowed limit, or it's considered as modification of messages
    and forbidden by some document?

    Fidonet Policy v.4.07

    ========= Here the quote begins ===========
    2.1.5 No Alteration of Routed Mail

    You may not modify, other than as required for routing or other technical purposes, any message, netmail or echomail, passing through the system from
    one FidoNet node to another.
    ========= Here it ends =============

    Michael

    ... node (at) f1042 (dot) ru
    --- GoldED+/LNX 1.1.5-b20170303
    * Origin: Moscow, Russia (2:5020/1042)
  • From Zhenja Kaliuta@2:4500/1.59 to Michael Dukelsky on Fri Jan 10 21:48:40 2020
    Hi, Michael!

    On Fri, 10 Jan 2020 21:40:01 +0200 Michael Dukelsky writes:

    In case of subject longer then allowed 71 + NUL bytes HPT send the
    whole PKT with the incorrect message to bads.

    Would it be more practical to be more error tolerant and just cut
    it to the allowed limit, or it's considered as modification of
    messages and forbidden by some document?

    Fidonet Policy v.4.07

    ========= Here the quote begins ===========
    2.1.5 No Alteration of Routed Mail

    You may not modify, other than as required for routing or other technical purposes, any message, netmail or echomail, passing through the system
    from
    one FidoNet node to another.
    ========= Here it ends =============

    Isn't it an "other technical purpose"? ;)


    --- Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
    * Origin: Somewhere in the North (2:4500/1.59)
  • From mark lewis@1:3634/12 to Zhenja Kaliuta on Fri Jan 10 19:51:55 2020
    Re: hpt long subjects and bad packets
    By: Zhenja Kaliuta to All on Fri Jan 10 2020 17:37:27


    In case of subject longer then allowed 71 + NUL bytes HPT
    send the whole PKT with the incorrect message to bads.

    what FTN software is creating and/or passing on messages with subject lines longer than the spec for packed messages allows??? whichever software that is needs to be fixed to stop the defective pkts from being generated and sent out in the first place...



    )\/(ark
    --- SBBSecho 3.10-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From Kai Richter@2:240/77 to Zhenja Kaliuta on Sat Jan 11 07:23:10 2020
    Hello Zhenja!

    10 Jan 20, Zhenja Kaliuta wrote to Michael Dukelsky:

    In case of subject longer then allowed 71 + NUL bytes HPT send
    the whole PKT with the incorrect message to bads.

    Would it be more practical to be more error tolerant and just
    cut it to the allowed limit, or it's considered as modification
    of messages and forbidden by some document?

    No. If you work around the bugs of others you will build a pile of shards.

    2.1.5 No Alteration of Routed Mail
    You may not modify, other than as required for routing or other
    technical purposes, any message, netmail or echomail, passing

    Isn't it an "other technical purpose"? ;)

    Work around a "standard violation" should never become a technical purpose.

    Fix the source of the problem. If it can't be fixed don't allow the broken to enter. That's straight, clear and simple.

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Tommi Koivula@2:221/1 to Zhenja Kaliuta on Sat Jan 11 09:53:00 2020
    10 Jan 20 21:48, Zhenja Kaliuta wrote to Michael Dukelsky:

    In case of subject longer then allowed 71 + NUL bytes HPT send the
    whole PKT with the incorrect message to bads.

    Would it be more practical to be more error tolerant and just cut
    it to the allowed limit, or it's considered as modification of
    messages and forbidden by some document?

    Fidonet Policy v.4.07

    ========= Here the quote begins ===========
    2.1.5 No Alteration of Routed Mail

    You may not modify, other than as required for routing or other technical
    purposes, any message, netmail or echomail, passing through the system
    from one FidoNet node to another.
    ========= Here it ends =============

    Isn't it an "other technical purpose"? ;)

    It depends who you ask. ;)

    I made a quick test: I set my terminal to 132x45 and entered a message to JAM base with GoldED with subject of 100 chars. It stayed there until "hpt scan" when it was cut to 72 chars. However this happened in my msgbase, not in transit mail.

    So it would be best to fix fidogate (?) not to send out illegal pkt's.

    'Tommi

    ---
    * Origin: 2001:470:1f15:cb0:f1d0:2:221:1 (2:221/1)
  • From Zhenja Kaliuta@2:4500/1.59 to Mark lewis on Sat Jan 11 09:49:17 2020
    Hi, mark!

    On Fri, 10 Jan 2020 19:51:55 +0200 mark lewis writes:

    In case of subject longer then allowed 71 + NUL bytes HPT
    send the whole PKT with the incorrect message to bads.

    what FTN software is creating and/or passing on messages with
    subject lines longer than the spec for packed messages allows???
    whichever software that is needs to be fixed to stop the defective
    pkts from being generated and sent out in the first place...

    No doubts. No arguments against that.

    But software may have bugs and it's sad to lose otherwise valid data
    because of that.

    But I understand the general attitude, thanks you for the input.

    BTW, in RFC world it's pretty different. Ex.: rfc2045

    ```
    (5) Encoded lines must not be longer than 76 characters,
    not counting the trailing CRLF. If longer lines are
    found in incoming, encoded data, a robust
    implementation might nevertheless decode the lines, and
    might report the erroneous encoding to the user.
    ```
    which looks more human friendly for me.

    And the original question can be splitted basically into 2:

    - is it ok to handle the situation without failing the pkt?
    - is it ok to make the pkt standard complied?


    I got your answers, thanks a lot!


    --- Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
    * Origin: Somewhere in the North (2:4500/1.59)
  • From Zhenja Kaliuta@2:4500/1.59 to Kai Richter on Sat Jan 11 09:54:05 2020
    Hi, Kai!

    On Sat, 11 Jan 2020 07:23:10 +0200 Kai Richter writes:

    In case of subject longer then allowed 71 + NUL bytes HPT send
    the whole PKT with the incorrect message to bads.

    Would it be more practical to be more error tolerant and just
    cut it to the allowed limit, or it's considered as modification
    of messages and forbidden by some document?

    No. If you work around the bugs of others you will build a pile of
    shards.

    Well, in general in the world there are other opinions:

    https://tools.ietf.org/html/rfc1122

    ```
    1.2.2 Robustness Principle

    [...]

    "Be liberal in what you accept, and
    conservative in what you send"
    ```

    https://tools.ietf.org/html/rfc7103

    ```Advice for Safe Handling of Malformed Messages```

    2.1.5 No Alteration of Routed Mail
    You may not modify, other than as required for routing or other
    technical purposes, any message, netmail or echomail, passing

    Isn't it an "other technical purpose"? ;)

    Work around a "standard violation" should never become a technical purpose.

    Valid point, thanks.

    Fix the source of the problem. If it can't be fixed don't allow the
    broken to enter. That's straight, clear and simple.


    --- Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
    * Origin: Somewhere in the North (2:4500/1.59)
  • From Zhenja Kaliuta@2:4500/1.59 to Tommi Koivula on Sat Jan 11 10:52:03 2020
    Hi, Tommi!

    On Sat, 11 Jan 2020 09:53:00 +0200 Tommi Koivula writes:

    In case of subject longer then allowed 71 + NUL bytes HPT send
    the whole PKT with the incorrect message to bads.

    Would it be more practical to be more error tolerant and just
    cut it to the allowed limit, or it's considered as modification
    of messages and forbidden by some document?

    Fidonet Policy v.4.07

    ========= Here the quote begins ===========
    2.1.5 No Alteration of Routed Mail

    You may not modify, other than as required for routing or other technical
    purposes, any message, netmail or echomail, passing through the system
    from one FidoNet node to another.
    ========= Here it ends =============

    Isn't it an "other technical purpose"? ;)

    It depends who you ask. ;)

    I made a quick test: I set my terminal to 132x45 and entered a
    message to JAM base with GoldED with subject of 100 chars. It
    stayed there until "hpt scan" when it was cut to 72 chars. However
    this happened in my msgbase, not in transit mail.

    Agree, it's a different case.

    So it would be best to fix fidogate (?) not to send out illegal
    pkt's.

    Oh, just to avoid wrong conclusions -- the original question was not
    because of fidogate producing incorrect packets, in this place it
    behaves correctly.

    I mentioned it in connection with handling incoming packets. I compared
    it to what hpt is doing (found one problem actually) and started
    thinking of what would be the best way.


    --- Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
    * Origin: Somewhere in the North (2:4500/1.59)
  • From Andrew Kolchoogin@2:5020/290.22 to Michael Dukelsky on Sat Jan 11 10:47:44 2020
    Hi Michael!

    Friday January 10 2020 you wrote to Zhenja Kaliuta the following:

    As usual, FPD contradicts with itself. :)

    2.1.5 No Alteration of Routed Mail

    You may not modify ... echomail
    Echomail is not routed nowadays. It is a sort of _direct_ mail from echo uplink
    to echo downlink. :)

    Yours
    Andrew Kolchoogin.

    ... Those who sacrifices freedom for security deserves neither
    --- Pen on parchment
    * Origin: -= Castle of Silence =- (2:5020/290.22)
  • From mark lewis@1:3634/12 to Zhenja Kaliuta on Sat Jan 11 07:28:05 2020
    Re: Re: hpt long subjects and bad packets
    By: Zhenja Kaliuta to Mark lewis on Sat Jan 11 2020 09:49:17


    - is it ok to handle the situation without failing the pkt?
    - is it ok to make the pkt standard complied?

    in my personal and professional opinion, the answer to both questions is a resounding "yes" but with at least the following caveat...

    1. HPT is the first FTN processor to handle the defective pkt
    after the flawed (gating?) software has created it...

    once the defective pkts are already in the distribution stream, meaning another
    FTN tosser has processed it, then no... mark them as bad and set them aside with a proper logged reason...

    eg: foobar.pkt - packed msg #XXX - subject field too long

    or some such... possibly also record the identity of the processor that created
    the defective packet if that's not already being done... that info can be gathered from the packet header... one might also want to log the above when the processor repairs such a defective message and log an additional message stating the repair decision... logging bad pkts is generally already done and adding logging that the repair of message #xxx is being done is trivial...

    others may or may not feel the same way...

    i fully agree that this is a technical problem which is allowed for by current fidonet policy but also keep the above caveat in mind... other FTNs may have different policy, though, and that should also be considered... it may lead to the "repairing tosser" to perform this fix or sidelining based on the FTN the packets belong to which would basically mean a setting per FTN... possibly something like

    FTN zone(s): 1-4095
    FTN domain: xxxxxxxx
    Repair defective pkts: yes/no

    in the configuration settings... i'm thinking of this in this manner because if
    the pkt is only 3D/4D, then the domain info is not available and the decision would be made based on the zone number... if the pkt is 5D, then the domain info is available and the decision would be made on that... YMMV is applicable here, though...


    )\/(ark
    --- SBBSecho 3.10-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From mark lewis@1:3634/12 to Kai Richter on Sat Jan 11 08:16:43 2020
    Re: hpt long subjects and bad packets
    By: Kai Richter to Zhenja Kaliuta on Sat Jan 11 2020 07:23:10


    Isn't it an "other technical purpose"? ;)

    Work around a "standard violation" should never become a technical
    purpose.

    Fix the source of the problem. If it can't be fixed don't allow the broken
    to enter. That's straight, clear and simple.

    keeping in mind my previous message on this, i also highly agree with this and still look to the caveat i listed in that previous message...

    ew (TINW) really need to know the name of this defective software creating pkts
    with too long subject lines... it is possible there's a newer/older version that doesn't have this defect and the originating system can be pointed to this
    version to replace their defective one...


    )\/(ark
    --- SBBSecho 3.10-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From mark lewis@1:3634/12 to Andrew Kolchoogin on Sat Jan 11 08:36:10 2020
    Re: hpt long subjects and bad packets
    By: Andrew Kolchoogin to Michael Dukelsky on Sat Jan 11 2020 10:47:44


    As usual, FPD contradicts with itself. :)

    there is a huge difference between technical specification and network policy... what was quoted was policy... FTS-0001 is a little confusing in its description of field formats in some cases but section A4 seems to be missed/ignored/skipped over by a lot of people...

    ----->8 snip 8<-----

    A. Introduction

    [...]

    4. Data Description

    A language specific notation was avoided. Please help stamp out
    environmental dependencies. Only you can prevent PClone market
    dominance. Don't panic, there are rectangular record layouts too.

    (* non-terminals *)
    UpperCaseName - to be defined further on

    (* literals *)
    "ABC" - ASCII character string, no termination implied
    nnH - byte in hexadecimal

    (* terminals *)
    someName - 16-bit integer, low order byte first (8080 style)
    someName[n] - field of n bytes
    someName[.n] - field of n bits
    someName(n) - Null terminated string allocated n chars (incl Null)
    someName{max} - Null terminated string of up to max chars (incl Null)

    (* punctuation *)
    a b - one 'a' followed by one 'b'
    ( a | b ) - either 'a' or 'b', but not both
    { a } - zero or more 'a's
    [ b ] - zero or one 'b'
    (* comment *) - ignored

    (* predeclared constant *)
    Null = 00H

    ----->8 snip 8<-----

    so, if the subject field of a packed message is defined as

    +-----------------------+-----------------------+
    | subject |
    ~ max 72 bytes ~
    | null terminated |
    +-----------------------+-----------------------+

    that means 71 bytes plus the terminating 0x00 byte...

    a lot of us know this but hobby coders may, as mentioned above, miss/ignore/skip section A4 and cause a defect in their code... the rest of us can either work around it as many have done over the years or we can reject those defects and bring the problem to the coder's attention...


    )\/(ark
    --- SBBSecho 3.10-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From Zhenja Kaliuta@2:4500/1.59 to Mark lewis on Sat Jan 11 16:17:23 2020
    Hi, mark!

    On Sat, 11 Jan 2020 08:16:43 +0200 mark lewis writes:

    Isn't it an "other technical purpose"? ;)

    Work around a "standard violation" should never become a technical
    purpose.

    Fix the source of the problem. If it can't be fixed don't allow
    the broken to enter. That's straight, clear and simple.

    keeping in mind my previous message on this, i also highly agree
    with this and still look to the caveat i listed in that previous message...

    Thanks for the detailed reply.

    ew (TINW) really need to know the name of this defective software
    creating pkts with too long subject lines... it is possible there's
    a newer/older version that doesn't have this defect and the
    originating system can be pointed to this version to replace their defective one...

    There is no problem there (anymore). It was some newly written software
    and the author is aware of it and has fixed it AFAIU.



    --- Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
    * Origin: Somewhere in the North (2:4500/1.59)
  • From Oli@2:280/464.47 to Kai Richter on Sat Jan 11 18:56:44 2020
    Fix the source of the problem. If it can't be fixed don't allow the
    broken to enter. That's straight, clear and simple.

    Maybe we should block all PKTs generated by Mystic BBS then? And also PKTs from
    hpt itself? (which doesn't set the JAM OADDRESS and rewrites the message date for Squish messages.



    * Origin: kakistocracy (2:280/464.47)
  • From Kai Richter@2:240/77 to Zhenja Kaliuta on Sun Jan 12 20:22:20 2020
    Hello Zhenja!

    11 Jan 20, Zhenja Kaliuta wrote to Mark lewis:

    But software may have bugs and it's sad to lose otherwise valid data because of that.

    I don't want to accept that argument because there is no clear line what is "a goog bug" that needs to be corrected by others and "a bad bug" that causes more
    harm than being useful. And even "good bugs" would create a bunch of workarounds with the risk of interferring each other by time.

    It's like a water bucket with a small hole. You can workaround by adding water up to full and run faster to your destination. Or you can place your hand on the hole. With more holes you will have less water at the destination or you need more hands to cover the holes. What do you do if there is no water in the bucket when you arrived at the destination? Throw the bucket away and get a new
    one? Or at last start fixing the holes?

    BTW, in RFC world it's pretty different. Ex.: rfc2045

    If there is a better bucket throw away the old one and use the new one.

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Kai Richter@2:240/77 to Zhenja Kaliuta on Sun Jan 12 21:21:38 2020
    Hello Zhenja!

    11 Jan 20, Zhenja Kaliuta wrote to Kai Richter:

    Well, in general in the world there are other opinions:

    https://tools.ietf.org/html/rfc1122

    Other opinions of another world.

    I do understand why people would like to "improve" fidonet or "advance" FTS. But hey, that job is already done. We already got rid of the numeric address system and use letters and numbers together, we can display pictures and videos
    next to text, we reduced the transfer time and many new standards where invented, even the name was refreshed to "The internet".

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Kai Richter@2:240/77 to mark lewis on Mon Jan 13 11:56:16 2020
    Hello mark!

    11 Jan 20, mark lewis wrote to Zhenja Kaliuta:

    - is it ok to handle the situation without failing the pkt?
    - is it ok to make the pkt standard complied?

    in my personal and professional opinion, the answer to both questions
    is a resounding "yes" but with at least the following caveat...

    Sorry, i didn't noticed your next mail where you highly agreed with me before writing. Please ignore this mail. I send it because i added some details of the
    consequences how works arounds can block each other.

    1. HPT is the first FTN processor to handle the defective pkt
    after the flawed (gating?) software has created it...

    The first FTN processor is the flawed software that generates non FTS pkts.
    No matter what that software normally does - it has a module that exports FTN pkt and that job should be done in accordance to FTS.

    once the defective pkts are already in the distribution stream,
    meaning another FTN tosser has processed it, then no... mark them as
    bad and set them aside with a proper logged reason...

    The result is that only direct link systems are flooded with broken content. That could be a nice idea, but because of the actual practice for network stability improvement called multi-link routing, there are many "first HPT tosser" that would receive the broken pkt.

    This is a good example that work arounds do interfere with other work arounds. Your idea would work for a straight star network but not for a mesh type network.

    or some such... possibly also record the identity of the processor
    that created the defective packet if that's not already being done...
    one might also want to log the above when the processor repairs such
    a defective message and log an additional message stating the repair decision... logging bad pkts is generally already done and adding
    logging that the repair of message #xxx is being done is trivial...

    others may or may not feel the same way...

    Instead of all other network members waste their resources for logging, fixing and accepting broken subjects within the network only one system needs to find a better solution how to transfer too long subject lines into the standard pkt format. A short and simple way is to copy the full subject line into the message text and cut the subject short plus adding a timestamp. The timestamp is required if the subject line is edited in the too long part only. This change must be visible in the short line somehow to keep reply linking searching and sorting by subject line working as usual.

    i fully agree that this is a technical problem which is allowed for
    by current fidonet policy

    I don't think so. I'm not in details but it doesn't make sense. The FTS are the
    standard valid for any FTS based networks to keep the networks running without problems. The FTS sets the frame. If the subject line is limited any FTN software have to be compliance to that limit - or it is not a FTS software.

    The purpose of the "other technical problems" in the policy can't be used to transform the network into a trashcan and leave the responsibility to find useful stuff in the trash to us.

    The problem of non FTS conform subject lines is a problem out of the FTN/FTS responsibility because it's not a valid FTS pkt. Period. The decision of HPT to
    move that non standard content to bad is correct. The decision not to modifiy and not to route the broken content is the only way to tell the source of the problem that his software has a serious bug that needs to be fixed.

    basically mean a setting per FTN... possibly something like
    Repair defective pkts: yes/no

    That would make it more complex. Keep it simple. It's not your/our job to fix incoming trash. HPT should detect a broken pkt, log it and move it to bad. That's how the broken pkt was found so hpt already works as designed.

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Kai Richter@2:240/77 to mark lewis on Mon Jan 13 12:06:46 2020
    Hello mark!

    11 Jan 20, mark lewis wrote to Kai Richter:

    ew (TINW) really need to know the name of this defective software
    creating pkts with too long subject lines... it is possible there's a newer/older version that doesn't have this defect and the originating system can be pointed to this version to replace their defective
    one...

    Good point. Additionally we all know about the development status of very old FTN software. If that software can't be fixed then there may be an alternate software that does the same job.

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Kai Richter@2:240/77 to Zhenja Kaliuta on Mon Jan 13 12:14:46 2020
    Hello Zhenja!

    11 Jan 20, Zhenja Kaliuta wrote to Mark lewis:

    There is no problem there (anymore). It was some newly written
    software and the author is aware of it and has fixed it AFAIU.

    That's a positiv surprise. Wonders will never cease. ;-)

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Kai Richter@2:240/77 to Oli on Mon Jan 13 12:27:30 2020
    Hello Oli!

    11 Jan 20, Oli wrote to Kai Richter:

    Fix the source of the problem. If it can't be fixed don't allow
    the broken to enter. That's straight, clear and simple.

    Maybe we should block all PKTs generated by Mystic BBS then?

    Well, i'm emotionally affected so i'm the wrong person for judgement. If you would start a seriously vote i would mark the YES field. But thats because some
    mails from that software crash my terminal output to unreadable. I have to quit
    my editor then, do a terminal reset and start again carefully skipping the broken mail. That is very annoying sometimes.

    And also PKTs from hpt itself? (which doesn't set the JAM OADDRESS and rewrites the message date for Squish messages.

    Actually i have no clue if msgbase formats are part of the FTS. I never noticed
    the msgbase format of incoming mails. I can't see if you use JAM or squish. Anyway your mail is stored in squish on my system. That's why i can't see a reason to drop a bug report.

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Oli@2:280/464.47 to Kai Richter on Tue Jan 14 08:34:03 2020
    13 Jan 20 12:27, you wrote to me:

    Fix the source of the problem. If it can't be fixed don't allow
    the broken to enter. That's straight, clear and simple.

    Maybe we should block all PKTs generated by Mystic BBS then?

    Well, i'm emotionally affected so i'm the wrong person for judgement.
    If you would start a seriously vote i would mark the YES field. But
    thats because some mails from that software crash my terminal output
    to unreadable. I have to quit my editor then, do a terminal reset and start again carefully skipping the broken mail. That is very annoying sometimes.

    I'm emotionally affected too. Most of the time I'm reading fido mails on my 4.8" phone (ssh session to my headless Raspberry). The screen is a bit small to
    display 80 columns in a comfortable font size. Mystic's hard wrapping of all mail that passes through to 80 chars/line is very annoying.

    And also PKTs from hpt itself? (which doesn't set the JAM OADDRESS
    and rewrites the message date for Squish messages.

    Actually i have no clue if msgbase formats are part of the FTS. I
    never noticed the msgbase format of incoming mails. I can't see if you
    use JAM or squish. Anyway your mail is stored in squish on my system. That's why i can't see a reason to drop a bug report.

    As far as I understand it from the documentation hpt always does import/export in one pass, is that correct? But What about echoareas that get rescanned? Around 50% of the mails (or a little bit less) will have a modified timestamp (DOS time format with 2 second granularity). To be clear, Squish message base format has a field for storing the original FTN time field, it is just not used
    by hpt (if I understand the C sources correctly).


    * Origin: kakistocracy (2:280/464.47)
  • From Michael Dukelsky@2:5020/1042 to Oli on Tue Jan 14 12:12:18 2020
    Hello Oli,

    Tuesday January 14 2020, Oli wrote to Kai Richter:

    And also PKTs from hpt itself? (which doesn't set the JAM
    OADDRESS and rewrites the message date for Squish messages.

    As far as I understand it from the documentation hpt always does import/export in one pass, is that correct? But What about echoareas
    that get rescanned? Around 50% of the mails (or a little bit less)
    will have a modified timestamp (DOS time format with 2 second granularity). To be clear, Squish message base format has a field for storing the original FTN time field, it is just not used by hpt (if I understand the C sources correctly).

    Could you please send me a bug report with full description of actions to reproduce the error with an example of the message base and pkt taking part in the actions. I am very busy at the moment but I'll try to investigate it when time permits.

    Michael

    ... node (at) f1042 (dot) ru
    --- GoldED+/LNX 1.1.5-b20170303
    * Origin: Moscow, Russia (2:5020/1042)
  • From Oli@2:280/464.47 to Michael Dukelsky on Tue Jan 14 15:12:47 2020
    And also PKTs from hpt itself? (which doesn't set the
    JAM OADDRESS and rewrites the message date for Squish
    messages.

    As far as I understand it from the documentation hpt
    always does import/export in one pass, is that correct?
    But What about echoareas that get rescanned? Around 50%
    of the mails (or a little bit less) will have a modified
    timestamp (DOS time format with 2 second granularity). To
    be clear, Squish message base format has a field for
    storing the original FTN time field, it is just not used
    by hpt (if I understand the C sources correctly).

    Could you please send me a bug report with full
    description of actions to reproduce the error with an
    example of the message base and pkt taking part in the
    actions. I am very busy at the moment but I'll try to
    investigate it when time permits.

    I'm not using hpt and I cannot provide example files, but the problem is simple:

    a squish tosser should write the date string to the __ftsc_date field when importing mails and should use the __ftsc_date field (if it is not empty) on export. see the squish spec. smapi seems to support that field, but hpt doesn't
    use it.

    the JAM problem:
    it was mentioned in some other echoarea that hpt doesn't set the OADDRESS for echomail and that the origin address is missing on rescans. i also couldn't find a place in the sources where the OADDRESS is set.

    jammnntpd even has a workaround for missing OADDRESS fields.

    please correct me if i'm wrong. if i had the time, i would do some testing with
    hpt.

    and thank you for your hpt support and bug fixing, michael.



    * Origin: kakistocracy (2:280/464.47)
  • From Michael Dukelsky@2:5020/1042 to Oli on Tue Jan 14 17:25:40 2020
    Hello Oli,

    Tuesday January 14 2020, Oli wrote to Michael Dukelsky:

    a squish tosser should write the date string to the __ftsc_date field
    when importing mails and should use the __ftsc_date field (if it is
    not empty) on export. see the squish spec. smapi seems to support that field, but hpt doesn't use it.

    the JAM problem:
    it was mentioned in some other echoarea that hpt doesn't set the
    OADDRESS for echomail and that the origin address is missing on
    rescans. i also couldn't find a place in the sources where the
    OADDRESS is set.

    jammnntpd even has a workaround for missing OADDRESS fields.

    OK, thank you, I'll look at it when I have time.

    Michael

    ... node (at) f1042 (dot) ru
    --- GoldED+/LNX 1.1.5-b20170303
    * Origin: Moscow, Russia (2:5020/1042)
  • From Tommi Koivula@2:221/1 to Oli on Tue Jan 14 19:53:30 2020
    14 Jan 20 15:12, Oli wrote to Michael Dukelsky:

    I'm not using hpt and I cannot provide example files, but the problem is simple:

    a squish tosser should write the date string to the __ftsc_date field when importing mails and should use the __ftsc_date field (if it is not empty)
    on
    export. see the squish spec. smapi seems to support that field, but hpt
    doesn't
    use it.

    Which field is that? Can I see it in GoldED hexdump pressing "i" ?

    These date's are found in your message in my squish base, tossed by hpt.

    === Cut ===
    DateTime : 14 Jan 20 15:12:47
    OrigAddr : 2:280/464.47
    DestAddr : 2:221/1.0
    See : 92, 0, 0, 0, 0, 0, 0, 0, 0
    Attr : 00020000h (00000000000000100000000000000000b)
    DateWritten : 2020-01-14 15:12:46 (7997502Eh)
    DateArrived : 2020-01-14 16:13:52 (81BA502Eh)
    UTC-Offset : 0
    === Cut ===

    'Tommi

    ---
    * Origin: 2001:470:1f15:cb0:f1d0:2:221:1 (2:221/1)
  • From Oli@2:280/464.47 to Tommi Koivula on Tue Jan 14 21:24:09 2020
    a squish tosser should write the date string to the
    __ftsc_date field when importing mails and should use the
    __ftsc_date field (if it is not empty) on export. see the
    squish spec. smapi seems to support that field, but hpt
    doesn't use it.

    Which field is that? Can I see it in GoldED hexdump
    pressing "i" ?

    Good question. I only discovered "i" recently by hitting the wrong key and don't know much about it ...

    These date's are found in your message in my squish base,
    tossed by hpt.

    ... but DateTime looks exactly like the value __ftsc_date should have. So I guess I was at least partly wrong. hpt does write __ftsc_date.

    === Cut ===
    DateTime : 14 Jan 20 15:12:47
    OrigAddr : 2:280/464.47
    DestAddr : 2:221/1.0
    See : 92, 0, 0, 0, 0, 0, 0, 0, 0
    Attr : 00020000h
    (00000000000000100000000000000000b)
    DateWritten : 2020-01-14 15:12:46 (7997502Eh)
    DateArrived : 2020-01-14 16:13:52 (81BA502Eh)
    UTC-Offset : 0
    === Cut ===



    * Origin: kakistocracy (2:280/464.47)
  • From Kai Richter@2:240/77 to Oli on Wed Jan 15 01:59:12 2020
    Hello Oli!

    14 Jan 20, Oli wrote to Kai Richter:

    if you use JAM or squish. Anyway your mail is stored in squish on
    my system. That's why i can't see a reason to drop a bug report.

    As far as I understand it from the documentation hpt always does import/export in one pass, is that correct?

    I don't know. There is control for import by "hpt toss" and control for export via "hpt scan". But for routing i don't know if it's possible to import pkt to local msgbase without exporting the mail to all linked systems. That option doesn't make sense to me. If the user deletes mails before they are forwarded to the linked systems that would be censorship and against the "no modification
    of in transit mail" of the policy.

    But i found another function that reveals that the date field is common for modification:

    ***
    @item Syntax:
    @code{processPkt <string>}
    @item Example:
    @code{processPkt pktdate}

    Space char and name of the current file "*.pkt" will be append into end of <string> and <string> willl be executed before tossing each pkt file. You
    are able to fix your pkts using pktdate or any other tool before tossing
    them. Note that pkt file may be renamed depending of @code{tossingExt}
    token value.
    ***

    In the past when i used OS/2 there was a tool called pktsort that could be hook
    at the same place to sort the mails by date. That fixed the situation that the answer was visible before the question in the msgbase.

    For hpt scan there is an option to bypass netmail:

    'When PackNetMailOnScan is "on" (default) hpt packs netmail when doing
    "hpt scan" and netmail area is found in EchoTossLog file. When it is
    "off" hpt leaves netmail area(s) in EchoTossLog file until "hpt pack" is invoked.'

    But What about echoareas that get rescanned? Around 50% of the mails
    (or a little bit less) will have a modified timestamp (DOS time format
    with 2 second granularity).

    That "adjusted" timestamp is common to me since i use fidonet. I don't know the
    decision for that, maybe it's due to 8-bit limitation of the date format on early fidonet computers back in the 80's?

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)