Just curious.

Allen
--- SBBSecho 3.00-Win32
* Origin: LiveWire BBS - Synchronet - LiveWireBBS.com (1:2320/100)

18 Feb 17 23:05:02, you wrote to All:

@TZUTC: -0500
@MSGID: 174.fn_utf8@1:2320/100 1d0e501c
@PID: Synchronet 3.17a-Win32 Feb 7 2017 MSC 1800
@TID: SBBSecho 3.00-Win32 r3.28 Feb 7 2017 MSC 1800
Just curious.

No CHRS klugde. What do you think? :D

'Tommi

... \\HAL has been up for: 92 day(s), 10 hour(s), 31 minute(s), 46 second(s) ---
* Origin: Point One. Ylöjärvi, Finland. (2:221/1.1)

Hello Allen!

18 Feb 17 23:05, you wrote to All:

Just curious.

No, only Michiel and Maurice. ;)

Kees

--- GoldED+/LNX 1.1.5
* Origin: As for me, all I know is that, I know nothing. (2:280/5003.4)

-={ 2017-02-19 10:11:16.483348140+00:00 }=-

Hey Tommi!

No CHRS klugde. What do you think? :D

Mitä väliä sillä on? :::evil grin:::

Life is good,
Maurice

... Se forholena cræft and forhyded gold ne bið ællunga ungelice.
Concealed skill and hidden gold are not entirely unalike.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

-={ 2017-02-19 11:32:59.417475392+01:00 }=-

Hey Kees!

No, only Michiel and Maurice. ;)

Out of those two only Maurice although Michiel has been trying which is worth something ... maybe.

Also I believe I was the first.

Also, also I was and am still the best. :-)

There are a few who can but for whatever reason chose to either shoot themselves in the foot by crippling their working apps with a seriously flawed "standard" or have chosen to remain quiet on the issue. Either/or I believe that still makes me the best given that it is and has been demonstrated to be accurate.

What's your excuse?

Life is good,
Maurice

... Sorh cymeð manig ond mislic in manna dream.
Sorrow comes in many and various ways amid the joys of men.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Re: Is Synchronet UTF8 Ready?
By: Tommi Koivula to Allen Prunty on Sun Feb 19 2017 11:39 am

No CHRS klugde. What do you think? :D

Obviously Tommi but am I missing a setting to turn it on?

Allen
--- SBBSecho 3.00-Win32
* Origin: LiveWire BBS - Synchronet - LiveWireBBS.com (1:2320/100)

Maurice Kinal wrote:

-={ 2017-02-19 10:11:16.483348140+00:00 }=-

Hey Tommi!

No CHRS klugde. What do you think? :D

Mitä väliä sillä on? :::evil grin:::

Niinpä niin, niinpä niin... ;)

Life is good,
Maurice

... Se forholena cræft and forhyded gold ne bið ællunga ungelice.
Concealed skill and hidden gold are not entirely unalike.

'Tommi

---
* Origin: rbb soupgate, the nntp echomail junction at (2:221/360.3001)

Hello Maurice!

19 Feb 17 10:32, you wrote to me:

What's your excuse?

Within the languages that I can speak, I can sufficiently express myself
with an ASR33. A mix of upper and lowercase is just a addition for
readabillity.

A few years ago I started a proof of concept in Python for a character
encoding smart message reader. It uses somthing like uchardet and some
additional tweaking to define a plausible enoding of the textblock that I
access. Subsequenly is uses iconvert to convert to UTF-8.

I does a fair job at making all messages presentable as they should, with
the proper glyphs. So some would be thrilled, but to me it is no use,
because I cannot read (understand) the text anyway. But it look much
nicer.

One of the reasons I did not continue, is the access to the message store.
Currently it can extract messages from squish. But I cannot store messages.

I could solve it the way, I think you use, by builing a PKT and use the
tosser to cope with the message store.

I think in my case Jamm as a message store would be more feasible, but there
are so many other things to do, that are more fun than writing code to
read and write to a Jamm message base.

Well I could take a easy intermediate, use *.msg.

The message to you, is I can properly present whatever you write, but I may
not be able to understand it. If I give an answer it will most likely be
encoded in Latin-1 of ISO-8859-1.

Whenever I feel the urge to write UTF-8 encoded messages, you and Michiel
will be the first to receive the crappy results. ;)

Some day this may happen if I can complete it in this life. Don't hold
your breath.

Kees

--- GoldED+/LNX 1.1.5
* Origin: As for me, all I know is that, I know nothing. (2:280/5003.4)

-={ 2017-02-19 17:51:31.227187588+02:00 }=-

Hey Tommi!

Niinpä niin, niinpä niin... ;)

I trust that it didn't matter given that it looks good despite your CHRS kludge.

Life is good,
Maurice

... Sprec ofter embe oðres monnes weldæde þonne emb ðine agna.
Speak more often about other people's good deeds than about your own.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

-={ 2017-02-19 17:06:40.652942563+01:00 }=-

Hey Kees!

So some would be thrilled, but to me it is no use, because I
cannot read (understand) the text anyway. But it look much
nicer.

Yes. I can relate. However that makes this much more of a challenge.

Whenever I feel the urge to write UTF-8 encoded messages, you
and Michiel will be the first to receive the crappy results. ;)

I am honoured.

Some day this may happen if I can complete it in this life.
Don't hold your breath.

Heeft de boerderij op het niet inzetten. :-)

Life is good,
Maurice

... Meotud ana wat hwær se cwealm cymeþ, þe heonon of cyþþe gewiteþ.
Only God knows where plague goes when it departs from a place.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

19 Feb 17 10:11, Maurice Kinal wrote to Tommi Koivula:

-={ 2017-02-19 10:11:16.483348140+00:00 }=-

Hey Tommi!

No CHRS klugde. What do you think? :D

Mitä väliä sillä on? :::evil grin:::

Sorry, my native codepage is not 850. :)

'Tommi

--- GoldED+/EMX 1.1.5-b20161221
* Origin: 2001:470:1f15:cb0:f1d0:2:221:0 (2:221/0)

Hello Maurice!

19 Feb 17 16:06, you wrote to me:

Some day this may happen if I can complete it in this life.
Don't hold your breath.

Heeft de boerderij op het niet inzetten. :-)

Daar kan ik geen chocola van maken.

Kees

--- GoldED+/LNX 1.1.5
* Origin: As for me, all I know is that, I know nothing. (2:280/5003.4)

Hello Maurice!

19 Feb 17 16:06, you wrote to me:

Some day this may happen if I can complete it in this life.
Don't hold your breath.

Heeft de boerderij op het niet inzetten. :-)

After thinking about the above sentence I think you are talking about
"betting the farm"

I cannot think of an equivalent Dutch saying at the moment. Besides I have
to think of one with weird characters that are only available in UTF8.

Without such provisions I will get a warning from Michiel.

Kees

--- GoldED+/LNX 1.1.5
* Origin: As for me, all I know is that, I know nothing. (2:280/5003.4)

-={ 2017-02-19 20:53:26.245843731+02:00 }=-

Hey Tommi!

Sorry, my native codepage is not 850. :)

Neither is mine. I never did nor do IBM. Before utf-8 messaging it was straight and honest 7 bit ascci and for the mostpart still is.

Life is good,
Maurice

... Eorþan indryhto ealdað ond searað, swa nu monna gehwylc.
The glory of earth ages and grows sere, as now every man does.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

-={ 2017-02-19 19:56:42.632225714+01:00 }=-

Hey Kees!

Daar kan ik geen chocola van maken.

A Møøse once bit my sister ...

Life is good,
Maurice

... Ræd biþ nyttost, yfel unnyttost.
Good advice is the most useful thing, bad the least useful.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

-={ 2017-02-19 19:59:13.459736274+01:00 }=-

Hey Kees!

I think you are talking about "betting the farm"

I think "not betting the farm" but my Dutch isn't that good ... to say the least.

Without such provisions I will get a warning from Michiel.

Probably. The old english utf-8 taglines are my lame way around that issue. I took the liberty of adding the modern english translation which probably isn't kosher for taglines but what the heck. I think it is worth the risk.

Life is good,
Maurice

... Blind byþ bam eagum se þe breostum ne starat.
He is blind in both eyes who does not look with the heart.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Konnichi wa, *Maurice-kun*! Aogu manako oyobi uketamawaru waga koe!
Tomodachi _Maurice Kinal_ tsukuru airon _Tommi Koivula_
Nichiji - /*19 ��� 17 15:51*/, Daizai - /*Re: Is Synchronet UTF8 Ready?*/:

I trust that it didn't matter given that it looks good despite your
CHRS kludge.

Actually, his CHRS kludge is perfectly valid.
On contrary, your messages don't contain any information about used codepage at all. So don't be surprised when they turns to complete garbage because heuristic detection makes a mistake.

Ganbatte, *Maurice*!

[_N0SF3R@TU_]
... GoldED-NSF/LNX 1.1.5-b20140107 (Linux 3.19.1-gentoo iF6M42)
--- #[Kaori Sekken: Master.NoSFeRaTU[@]Gmail.com] [Kumi Nyaa]#
* Origin: Ojisan, oriru mottekuru suna oyobi korosu sagaru kabe (2:5019/40)

-={ 2017-02-27 17:47:59.185178113+00:00 }=-

Hey Konstantin!

Actually, his CHRS kludge is perfectly valid.

But does it matter? Is there any way to verify it being accurate? For example if I claim CP866 characters in my message by adding a CHRS that claims it, how do you or anyone else confirm that it indeed contains CP866 and not some other 8 bit codes?

On contrary, your messages don't contain any information about
used codepage at all.

echo "Hafa ðu geleafa to lifes frumon; gewuna þar ðu wunodest." | file - /dev/stdin: UTF-8 Unicode text

:-)

So don't be surprised when they turns to complete garbage because heuristic detection makes a mistake.

That is never the problem with actual utf-8 text. The problem is when an inferior editor quotes them back.

Life is good,
Maurice

... Hafa ðu geleafa to lifes frumon; gewuna þar ðu wunodest.
Have faith in the creator of life; dwell where you have dwelt.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Konnichi wa, *Maurice-kun*! Aogu manako oyobi uketamawaru waga koe!
Tomodachi _Maurice Kinal_ tsukuru airon _Konstantin Kuzov_
Nichiji - /*27 ��� 17 17:47*/, Daizai - /*Is Synchronet UTF8 Ready?*/:

Actually, his CHRS kludge is perfectly valid.

But does it matter?

Yes it is. This is the only reliable source of actual used codepage.

Is there any way to verify it being accurate? For example if I claim
CP866 characters in my message by adding a CHRS that claims it, how do
you or anyone else confirm that it indeed contains CP866 and not some other 8 bit codes?

It doesn't need to be verified in any way. If presented kludge doesn't match actual used codepage then writer of that message hasn't configured his editor right.

So don't be surprised when they turns to complete garbage because
heuristic detection makes a mistake.

That is never the problem with actual utf-8 text. The problem is when
an inferior editor quotes them back.

The point is: if you doesn't present codepage information then recepient may see an incorrectly decoded message despite the fact that his editor supports unicode. Assumption that any message without CHRS kludge is UTF-8 by default is completely wrong, especially in fidonet where dominating codepage is CP866. And you can't rely completely on hereustic analysis algorithms as they all have margin of error.

I had written about that in this echo in the past: area://UTF-8?msgid=2:5019/40+53b49cb9

My position didn't changed since then:
=== QUOTE ===
if anyone write message with bogus or absent CHRS kludge then he is enemy to himself! Recepients can't reliable identify codepage in automatic mode. So, possibly they can't read this message at all, not because their soft is outdated and not supporting this codepage, but because author of the message don't indicate what codepage he is actually writing in and software isn't smart enough to successfully guess.
=== QUOTE ===

Ganbatte, *Maurice*!

[_N0SF3R@TU_]
... GoldED-NSF/LNX 1.1.5-b20140107 (Linux 3.19.1-gentoo iF6M42)
--- #[Kaori Sekken: Master.NoSFeRaTU[@]Gmail.com] [Kumi Nyaa]#
* Origin: Ojisan, oriru mottekuru suna oyobi korosu sagaru kabe (2:5019/40)

-={ 2017-02-27 19:56:45.307386294+00:00 }=-

Hey Konstantin!

This is the only reliable source of actual used codepage.

I disagree. It is probably the most unreliable source along with other misinformation that I won't get into in this particular echo but could easily atand as evidence why the above quote is disagreeable.

It doesn't need to be verified in any way.

True. However if only ascii and utf-8 text messages were the only supported encodings then verification is indeed possible and as a rule highly accurate whereas with 8 bit encodings all bets are off and verification is a waste of time as well as unneeded. Same with CHRS being a total waste of bytes as it means nothing and does nothing ... or at least *should* do nothing.

especially in fidonet where dominating codepage is CP866

That doesn't make it right. In fact I think it is a very good arguement as well as evidence as to what is wrong in fidonet.

My position didn't changed since then

That is because we have discussed this before. Now that you have been shown the errors of your ways I am positive that your stance is in dire need of adjustment. :::evil grin:::

Life is good,
Maurice

... Forst sceal freosan, fyr wudu meltan, eorþe growan, is brycgian.
Frost must freeze, fire melt wood, earth grow, ice form bridges.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Greetings, Maurice!

I disagree. It is probably the most unreliable source along with other misinformation that I won't get into in this particular echo but could easily atand as evidence why the above quote is disagreeable.

It is the ONLY currently available and more importantly already widespreed source. There are nothing else currently and I doubt will ever be.
Are you don't like currently used standard with "level" legacy? Make your own, implement that into all popular editors/gates, convince majority of fidonet users to switch. And we all will be happy. ^_^
But scrapping CHRS from messages isn't a solution, for example your current message clearly indicates that:

Google's compact_enc_det:
[+] Confident: NO
[+] Detected: GB18030
[+] Charset: GBK

Mozilla's uchardet:
[+] Charset: WINDOWS-1258

Extremely Naive Charset Analyser v1.18:
[+] Charset: NOT DETECTED >_<

Extremely Naive Charset Analyser GIT:
[+] Charset: UTF-8

Final result with WINDOWS-1258 you can see there: https://fido.g0x.ru/?area://UTF-8?msgid=1:153/7001+58b4847d

True. However if only ascii and utf-8 text messages were the only supported encodings then verification is indeed possible and as a rule highly accurate

If only ascii and utf-8 codepages were used then any verification is not needed at all, because ascii symbols have the same positions in utf-8. But that's not the case.
Situation for cyrillic users is actually worse as UTF-8 can't be used there at all currently. Because all unicode message will be completely ruined by non-unicode ready editor. Not just some insignificant portion of it. And wast majority of users still use these ancient editors (>90% as least).

BTW using codepages other than CP866 also still prohibited by rules in majority of the ex-ussr echos.

whereas with 8 bit encodings all bets are off and verification is a
waste of time as well as unneeded.

Again why it need to verified? It is author's job to provide all necessary technical information needed for properly processing his messages on receivers end. If he doesn't want to then he is "Сам себе злобный буратино".

Same with CHRS being a total waste of bytes as it means nothing and
does nothing ... or at least *should* do nothing.

Really? Miserable additional 10-15 bytes per message in 2017 is a big deal? Great argument... Delete your useless tagline and you are done. ^_^
And your "nothing" mantra exists only in your imagination and has nothing to do with reality.

especially in fidonet where dominating codepage is CP866

That doesn't make it right. In fact I think it is a very good arguement
as well as evidence as to what is wrong in fidonet.

What is right? All fidonet users must be using UTF-8 and only UTF-8? And what if they don't want to? Because you know... don't need to? They perfectly fine with current restricted codepage they are using for several decades already.
Also you said something about bytes, for cyrilic audience UTF-8 will bloat every message to almost twice in size.

My position didn't changed since then

That is because we have discussed this before. Now that you have been shown the errors of your ways

Which errors exactly? You weren't made a single argument, just a bunch of purposely garbaged messages and continue to produce messages whose prone to mojibake.

I am positive that your stance is in dire need of adjustment.
:::evil grin:::

Петросян...

--- Claws Mail 3.14.0 (GTK+ 2.24.30; x86_64-pc-linux-gnu)
* Origin: Via 2:5019/40 NNTP (GaNJaNET STaTi0N, Smolensk) (2:5019/40.1)

-={ 2017-02-28 14:19:00.123069135+00:00 }=-

Hey Konstantin!

Are you don't like currently used standard with "level" legacy?

My software doesn't require it. I don't see a need or use for it although I think I might 'understand' the original idea for it. I doubt anyone, other than maybe the original author, actually uses it.

convince majority of fidonet users to switch

I don't need to. The real world will take care of it much better than I can. I'd suggest that it already has.

Final result with WINDOWS-1258 you can see there

That doesn't look any worse than other crippleware I've seen out there being deployed in Fidonet. Anyhow the good stuff, such as 'file', shows it as UTF-8 which indeed it was. Same with this reply which is from the software I regularly use, including with Fidonet.

Life is good,
Maurice

... Blind byþ bam eagum se þe breostum ne starat.
He is blind in both eyes who does not look with the heart.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

-={ 2017-02-28 19:51:33.180210075+00:00 }=-

Hey Konstantin!

Extremely Naive Charset Analyser v1.18:
[+] Charset: NOT DETECTED >_<

Extremely Naive Charset Analyser GIT:
[+] Charset: UTF-8

I've never heard of this before now but it looks like the git version is on the right track. As for the message I am replying to (MSGID: 2:5019/40.1 58b5732e) when cat'ted to file-5.30 (the most recent version) reports; "/dev/stdin: UTF-8 Unicode text". It uses the magic file from /usr/share/misc/magic. Older versions made the same observations about UTF-8 text messages and I never once saw it fail ... unless of course it actually wasn't a UTF-8 text message and then it either reported, "data", in the case of a raw pkt or "Non-ISO extended-ASCII text" in the case of your message converted to CP866 and piped to file-5.30. Older versions of file often 'guessed' it wrong as ISO-8859-1 text.

What is right? All fidonet users must be using UTF-8 and only
UTF-8?

Given some more time it won't matter. I question that it already doesn't matter. I know for a fact that here at 1:153/7001 it doesn't matter and the the correct way is to make everything UTF-8 which eventually will be the way it will all be if indeed it already isn't. As for the CP437 Fidonet sysops who used to dominate far more than the CP866 Russian sysops do today, they pretty well much have either died off or have slipped off to obscurity haolding onto their abandonware with a death-like grip. DOS-think is at best just an oddity in Fidonet and seems to have temporarily found a home in Russia.

It is doomed. ;-)

Life is good,
Maurice

... Weorða ðe selfne godum dædum, ðenden ðin God recce.
Bring honour to yourself with good deeds, while God guides you.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

-={ 2017-02-28 21:46:51.486953722+00:00 }=-

Hey Konstantin!

Extremely Naive Charset Analyser GIT:
[+] Charset: UTF-8

------------------- <Esc>:r !enca --version starts
enca 1.20-dev

Features: -librecode-interface +iconv-interface +external-converter +language-detection -locale-alias +target-charset-auto +ENCAOPT

Copyright (C) 2000-2005 David Necas (Yeti) (<yeti@physics.muni.cz>),
2005 Zuxy Meng (<zuxy.meng@gmail.com>).

Enca is free software; it can be copied and/or modified under the terms of version 2 of GNU General Public License, run `enca --license' to see the full license text. There is NO WARRANTY; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
------------------- <Esc>:r !enca --version ends

For the message I am replying to, "MSGID: 2:5019/40.1 58b5732e", piped to 'enca -L none -' it reports;

Universal transformation format 8 bits; UTF-8

which is correct. However if I convert it to either KOI8-R, CP1251, ISO-8859-5 or CP866 then I have to specify the language "-L russian" for it to correctly identify the encoding. It did get it correct in all cases whereas file only identified it correctly when it is UTF-8 encoded but does properly identify it as 8 bit characters no matter which of KOI8-R, CP1251, ISO-8859-5 or CP866 was deployed.

Interesting.

Life is good,
Maurice

... Sibb æfre ne mæg wiht onwendan, þam ðe wel þenceð.
Kinship can never in any way be set aside, for one who thinks rightly.
--- GNU bash, version 4.4.12(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hello Мауриcе!

19 Feb 17 10:32, Мауриcе Кинал wrote to me:

-={ 2017-02-19 11:32:59.417475392+01:00 }=-

Hey Kees!

What's your excuse?

Шоулд И hаве оне.

Тhис ис лике РОТ13 ин АСИИ апплиед то УТФ-8

Ин реалиты ит ис уттер cрап

שָׁלוֹם

سلام

Kees

--- RaspberryPoint v0.1.141209 GoldED+/LNX 1.1.5--b20160201
* Origin: Some say UTF-8 is the solution, but is it? (2:280/5006.4)

Hello, Kees van Eeten.
On 08/03/2017 16:20 you wrote:

Hello Мауриcе!
19 Feb 17 10:32, Мауриcе Кинал wrote to me:

-={ 2017-02-19 11:32:59.417475392+01:00 }=-
Hey Kees!
What's your excuse?

Шоулд И hаве оне.
Тhис ис лике РОТ13 ин АСИИ апплиед то УТФ-8 Ин реалиты ит ис уттер cрап
שָׁלוֹם
سلام
Kees

Looks fine with hotdoged. But I dont understand any of it. ;)

---
* Origin: rbb soupgate, the nntp echomail junction at (2:221/360.3001)

Hello Tommi,

On Wednesday March 08 2017 20:19, you wrote to All:

What's your excuse?

Шоулд И hаве оне.
Тhис ис лике РОТ13 ин АСИИ апплиед то
УТФ-8 Ин реалиты ит ис уттер cрап
שָׁלוֹם سلام Kees

Looks fine with hotdoged. But I dont understand any of it. ;)

Run it through Google translate, but instead of looking at the translation om the right, look at the transliteration of Cyrillic to Latin below the left field.


Cheers, Michiel

--- GoldED+/W32-MINGW 1.1.5-b20110320
* Origin: Blijf Tønijn (2:280/5555)