Hello everybody.

I would like to start a trial of UTF-8 implementation for R80 with my system.

What would be the necessary steps? How can we receive and process the UTF-8 nodelists?
My main board runs Mystic, but I can setup an alternate system for the sake of the testing...

Flavio

... "Entra ali no segundo buraco" - Cris
--- MacFidoIP 1.0 (OSX)
* Origin: Hyperion's Orbit - Resisting since 1995! (4:801/189.1)

Hello Flavio,

On Thursday November 24 2016 09:43, you wrote to All:

I would like to start a trial of UTF-8 implementation for R80
with my system.

Great!

What would be the necessary steps? How can we receive and
process the UTF-8 nodelists?

1) Get a copy of the latest version of MakeNl. The latest version is 3.4.6

2) Get an UTF capable text editor. My favorite is de windows version of Vi. But Notepad will do as well.

3) Getting copies of some character conversion utilities like iconv is helpull but not a requirement. A serial editor like sed is also helpful.

My main board runs Mystic, but I can setup an alternate system
for the sake of the testing...

Mystic is poor in supporting anything other than pure ASCII. But to produce a UTF-8 nodelist segment you do not need a BBS.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20130111
* Origin: Blijf Tønijn (2:280/5555)

-={ 2016-11-24 14:05:16.427248174+00:00 }=-

Hey Michiel!

MvdV> 1) Get a copy of the latest version of MakeNl. The latest
MvdV> version is 3.4.6

---------- fetch and build makenl starts
wget https://sourceforge.net/projects/makenl/files/makenl/3.4.5/mn345src.zip unzip mn345src.zip
cd makenl-3.4.5/src/
make -f makefile.linux
install -sv ./makenl ~/bin
---------- fetch and build makenl ends

---------- <Esc>:read !makenl -h
MakeNL 3.4.5 (Linux) compiled with GNU C on Nov 24 2016 13:49:05
Usage: makenl [<config file>] [-p|-t] [-m[=<nodelist>]] [-n=<netname>] [-d]

-p (-process) Operate in Process mode, overriding Process statement
in makenl.ctl.

-t (-test) Operate in Test mode even on the normal process day.

-m (-merge) Merge your updated nodelist segment with an existing
distribution nodelist to produce a temporary nodelist
containing your updates.

-n (-name) Specialized cosmetic switch. When you make a composite
nodelist, the generated list begins, by default, with the
line 'Nodelist for ...'. The -name switch may be used to
insert a network name in front of the first word in that
line. If the specified replacement name contains spaces,
the entire parameter must be enclosed in quotes.

-d (-debug) Enable debugging output for some functions.

-c (-credits) Who made this possible.
---------- end

---------- <Esc>:read !ldd ~/bin/makenl | tr -d '\t'
linux-vdso.so.1 (0x00007ffc7a3cc000)
libc.so.6 => /lib/libc.so.6 (0x00007f1de85d0000)
/lib/ld-linux-x86-64.so.2 (0x00007f1de88fd000)
---------- end

MvdV> 2) Get an UTF capable text editor. My favorite is de windows
MvdV> version of Vi. But Notepad will do as well.

How about this instead as I lack a Windows box;

---------- <Esc>:read !vim --version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 21 2016 22:50:02)
Compiled by root@lilmikey
Normal version without GUI. Features included (+) or not (-):
-arabic +file_in_path -mouse_sgr +tag_binary
+autocmd +find_in_path -mouse_sysmouse +tag_old_static -balloon_eval +float -mouse_urxvt -tag_any_white
-browse +folding +mouse_xterm -tcl
+builtin_terms -footer +multi_byte +terminfo
+byte_offset +fork() +multi_lang +termresponse
+cindent -gettext -mzscheme +textobjects
-clientserver -hangul_input +netbeans_intg +title
-clipboard +iconv +path_extra -toolbar
+cmdline_compl +insert_expand -perl +user_commands
+cmdline_hist +jumplist +persistent_undo +vertsplit
+cmdline_info -keymap +postscript +virtualedit
+comments -langmap +printer +visual
-conceal +libcall -profile +visualextra
+cryptv +linebreak -python +viminfo
-cscope +lispindent -python3 +vreplace
+cursorbind +listcmds +quickfix +wildignore
+cursorshape +localmap +reltime +wildmenu
+dialog_con -lua -rightleft +windows
+diff +menu -ruby +writebackup
+digraphs +mksession +scrollbind -X11
-dnd +modify_fname +signs -xfontset
-ebcdic +mouse +smartindent -xim
-emacs_tags -mouseshape -sniff -xsmp
+eval -mouse_dec +startuptime -xterm_clipboard
+ex_extra -mouse_gpm +statusline -xterm_save
+extra_search -mouse_jsbterm -sun_workshop
-farsi -mouse_netterm +syntax
system vimrc file: "/etc/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -march=atom -m64 -g -Os -pipe -fPIC -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -Wl,--as-needed -o vim -lm -lncurses -lelf -lnsl -lacl -lattr -ldl
---------- end

MvdV> 3) Getting copies of some character conversion utilities like
MvdV> iconv is helpull but not a requirement. A serial editor like
MvdV> sed is also helpful.

---------- <Esc>:read !iconv --version
iconv (GNU libc) 2.24
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Ulrich Drepper.
---------- end

---------- <Esc>:read !sed --version
sed (GNU sed) 4.2.2
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <bug-sed@gnu.org>.
Be sure to include the word ``sed'' somewhere in the ``Subject:'' field. ---------- end

Now what? :::evil grin:::

Life is good,
Maurice

... Oft for ðæs lareowes unwisdome misfaraþ ða hieremenn.
Often because of the leader's folly, the followers go astray.
--- GNU bash, version 4.4.5(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hello Flavio,

Thursday November 24 2016 13:28, I wrote to you:

2) Get an UTF capable text editor. My favorite is de windows version
of Vi. But Notepad will do as well.

3) Getting copies of some character conversion utilities like iconv is helpull but not a requirement. A serial editor like sed is also
helpful.

Or... if you only need to enter accented and tilded characters and if you have iconv, you do not really need an utf-8 editor. You can use any editor capable of dealing with 8 bit character set. And then convert the CP850, Latin-1 or whatever encoded file to UTF-8 using iconv.


Cheers, Michiel


--- GoldED+/W32-MSVC 1.1.5-b20130111
* Origin: Blóf Tønón (2:280/5555)

Hello Flavio,

On 24 Nov 16 09:43, Flavio Bessa wrote to All:

Hello everybody.

I would like to start a trial of UTF-8 implementation for R80
with my system.

Excellent!

What would be the necessary steps? How can we receive and
process the UTF-8 nodelists?

Obviously your system needs to support UTF-8. I see by your tagline you're running OSX so that's probably true already. "locale" at a command line will definitely tell you, though.

As for the nodelist, Michiel van der Vlist is the guy to talk to about that. Receiving and processing that nodelist is OK, but you will want to do it separately from the original nodelist (and in the future only pass it on to other UTF-8 facilities), since other systems that do not support UTF-8 will not be able to process it correctly.

My main board runs Mystic, but I can setup an alternate system
for the sake of the testing...

You would definitely need to setup an alternate system for reading/writing UTF-8 messages in Fidonet. Mystic, while it supports local display in a UTF-8 environment, does not support UTF-8 in it's message bases yet.

If you use Golded, you will not be able to read some UTF-8 messages, but by using an external editor like nano or vim, once that loads you will be able to read and write in UTF-8, and Golded will not alter them if you setup your xlat stuff properly. If you wish to go this route, I'm sure one of us could help you set it up correctly (at least in Golded). The external editor would be completely up to you, though.. as long as it supports UTF-8.

Other than that, it's awesome to see activity in zone 4! Nice to meet you, Flavio.

Regards,
Nick

... "Не знаю. Я здесь только работаю."
--- GoldED+/LNX 1.1.5-b20160827
* Origin: thePharcyde_ distribution system (Wisconsin) (1:154/10)

Hello Maurice,

On 24 Nov 16 14:05, Maurice Kinal wrote to Michiel van der Vlist:

MvdV>> 1) Get a copy of the latest version of MakeNl. The latest
MvdV>> version is 3.4.6

---------- fetch and build makenl starts
wget https://sourceforge.net/projects/makenl/files/makenl/3.4.5/mn345s rc.zip unzip mn345src.zip cd makenl-3.4.5/src/ make -f makefile.linux install -sv ./makenl ~/bin
---------- fetch and build makenl ends

---------- <Esc>:read !makenl -h
MakeNL 3.4.5 (Linux) compiled with GNU C on Nov 24 2016 13:49:05

That should be v3.4.6. But I do believe v3.4.5 also supports UTF-8. The only change in the latest version was the removal of a BOM if it exists.

Regards,
Nick

... "Не знаю. Я здесь только работаю."
--- GoldED+/LNX 1.1.5-b20160827
* Origin: thePharcyde_ distribution system (Wisconsin) (1:154/10)

-={ 2016-11-24 10:52:44.931151035-06:00 }=-

Hey Nicholas!

That should be v3.4.6.

I don't see 3.4.6 at the site. Cutting and pasting from the site https://sourceforge.net/projects/makenl/files/makenl/ shows;

"Looking for the latest version? Download mn345src.zip (221.4 kB)"

But I do believe v3.4.5 also supports UTF-8.

I haven't read the documentation yet.

The only change in the latest version was the removal of a BOM if it exists.

Good idea. The BOM isn't needed in utf-8.

Life is good,
Maurice

... Styran sceal mon strongum mode; storm oft holm gebringeþ.
A strong mind must be steered; the sea often brings a storm.
--- GNU bash, version 4.4.5(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hello Maurice,

On 24 Nov 16 16:52, Maurice Kinal wrote to Nicholas Boel:

That should be v3.4.6.

I don't see 3.4.6 at the site. Cutting and pasting from the site https://sourceforge.net/projects/makenl/files/makenl/ shows;

"Looking for the latest version? Download mn345src.zip (221.4 kB)"

It may not have been updated to "stable" yet, which would probably be that packaged version button you're seeing. You may have to pull the current code directly from CVS (using the 'cvs' command, obviously).

Regards,
Nick

... "Не знаю. Я здесь только работаю."
--- GoldED+/LNX 1.1.5-b20160827
* Origin: thePharcyde_ distribution system (Wisconsin) (1:154/10)

Hello Maurice,

On Thursday November 24 2016 16:52, you wrote to Nicholas Boel:

The only change in the latest version was the removal of a BOM if
it exists.

Removing BOM's is an option.

Good idea. The BOM isn't needed in utf-8.

MakeNl is for making nodelists. Not only is a BOM not needed in a nodelist, it is not wanted because many nodelist processing softweres will see the presence of a BOM as an error condition.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20130111
* Origin: Blijf Tønijn (2:280/5555)

-={ 2016-11-25 03:00:58.185784795+00:00 }=-

Hey Michiel!

MvdV> many nodelist processing softweres will see the presence of a
MvdV> BOM as an error condition.

Which nodelist and what nodelist processing software? I believe it was clear that the current subject was related a utf-8 nodelist and not the distributed nodelist which is 7 bit ascii characters only, with exactly only three of which are C0 control codes. The BOM being three 8 bit bytes *should* produce an error for any nodelist processing software given that the BOM isn't 7 bit ascii. But then what does any of this have to do with creating a utf-8 nodelist?

The original question was (and I quote); "How can we receive and process the UTF-8 nodelists?" I don't believe you've properly answered this question yet.

Life is good,
Maurice

... Heald hordlocan, hyge fæste bind mid modsefan.
Hold close the treasure-chest, bind your thoughts fast within the heart. --- GNU bash, version 4.4.5(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hello Maurice,

On 25 Nov 16 03:00, Maurice Kinal wrote to Michiel van der Vlist:

The original question was (and I quote); "How can we receive and
process the UTF-8 nodelists?" I don't believe you've properly
answered this question yet.

At the moment I do not think anyone in Z1 is involved with this project, but I do plan on changing that (at least as far as receiving it here on my system when it's hatched). So to answer your question specifically to receive it, you can simply link to any of the Z2 systems currently involved, or since you're already linked here for some things, once I link to Michiel you can receive it from me if you wish to do so.

As for processing, your system would have no need to process anything (ie: with makenl itself). Since you're not an NC, RC, or ZC, you don't have to add anything to the current nodelist.

However, if Michiel and or others in Z2 who have setup the nodelist processing heirarchial (within the *C structure) just like the regular nodelist wouldn't mind, and we can not get our heirarchial (*C structure) side to join in.. maybe it would be allowed that you and I could process our own segments into the UTF-8 nodelist. Then and only then would you need to "process" anything at your end.

Regards,
Nick

... "Не знаю. Я здесь только работаю."
--- GoldED+/LNX 1.1.5-b20160827
* Origin: thePharcyde_ distribution system (Wisconsin) (1:154/10)

-={ 2016-11-25 10:03:20.826620583-06:00 }=-

Hey Nicholas!

once I link to Michiel you can receive it from me if you wish to
do so

Sounds good. I looked around everywhere I could think of without any luck other than some references that led nowhere significant. As for a compiler I doubt I really need one but would appreciate documentation as to valid characters as well as what fields multibyte characters are allowed (eg BBS name, sysop name, etc).

Then and only then would you need to "process" anything at your
end.

The only reason there would be a need is if and when any locals wish to replicate what I have done to keep Fidonet alive on anything I have any influence over. At the moment only one neighbour has ever expressed interest and I've tried to get him to use online BBS's instead but he insists on doing it my way-ish. I think he thinks he is better than me. The fool. ;-)

Anyhow he especially likes my old English random taglines and is very jealous which is probably why I keep them. They are utf-8 and need to be since the latin1 based 8 bit character sets lack some of the required characters. I am missing one but it never comes up and I don't think the Anglo-Saxons used it much (ever?).

Life is good,
Maurice

... Ne byð þæt fele freond, se þe oþrum facn heleð.
He who harbours treachery against another is not a faithful friend.
--- GNU bash, version 4.4.5(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hello Maurice,

On Friday November 25 2016 03:00, you wrote to me:

MvdV>> many nodelist processing softweres will see the presence of a
MvdV>> BOM as an error condition.

Which nodelist and what nodelist processing software?

Any nodelist and AFAIK all nodelist processing software except the latest version of MakeNl with the "REMOVEBOM 1" option set.

I believe it was clear that the current subject was related a utf-8 nodelist and not the distributed nodelist which is 7 bit ascii
characters only,

Then why bring up the 7 bit nodelist?

with exactly only three of which are C0 control codes. The BOM being three 8 bit bytes *should* produce an error for any nodelist
processing software given that the BOM isn't 7 bit ascii. But then
what does any of this have to do with creating a utf-8 nodelist?

Most nodelist processor software is not as shy as the old MakeNl. The old MakeNL will rigidly replace any non ASCII character with a question mark, but most other software is character encoding agnostic and will just pass them on "as is". This does not generate an error in the fields where "free text" is allowed.

The reason that a BOM causes an error is that a BOM normally appears at the start of a file, and therefore at the start of a line. And the start of a nodelist line is not "free text". The start of a line in the nodelist must be a semicolon, a comma or a valid keyword en a BOM is none of that.

The original question was (and I quote); "How can we receive and
process the UTF-8 nodelists?" I don't believe you've properly
answered this question yet.

This thread has not ended. I did not plan to give all the answers in one message. I just give the first steps.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20130111
* Origin: Blijf Tønijn (2:280/5555)

-={ 2016-11-26 00:58:09.381648128+00:00 }=-

Hey Michiel!

MvdV> This thread has not ended.

Okay. I'll stay out of the way now that I understand. Sorry for any confusion I may have caused.

Life is good,
Maurice

... Wineleas wonsælig mon genimeð him wulfas to geferan.
A friendless, unfortunate man takes wolves as his companions.
--- GNU bash, version 4.4.5(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hello Maurice,

On 26 Nov 16 00:58, Maurice Kinal wrote to Michiel van der Vlist:

Okay. I'll stay out of the way now that I understand. Sorry for any confusion I may have caused.

FYI, the DAILYUTF file echo is indeed available here now if you wish to access Filefix here via the same method you use for Areafix.

Regards,
Nick

... "Не знаю. Я здесь только работаю."
--- GoldED+/LNX 1.1.5-b20160827
* Origin: thePharcyde_ distribution system (Wisconsin) (1:154/10)

-={ 2016-11-25 22:58:24.146308534-06:00 }=-

Hey Nicholas!

the DAILYUTF file echo is indeed available

Thank you.

Life is good,
Maurice

... Ne wat swetes ðanc, se þe biteres ne onbyrgeð.
He never knows the pleasure of sweetness, who never tastes bitterness.
--- GNU bash, version 4.4.5(1)-release (x86_64-atom-linux-gnu)
* Origin: Little Mikey's Brain - Ladysmith BC, Canada (1:153/7001)

Hello Nicholas!

25 Nov 16 09:25, you wrote to Maurice Kinal:

However, if Michiel and or others in Z2 who have setup the nodelist processing heirarchial (within the *C structure) just like the regular nodelist wouldn't mind, and we can not get our heirarchial (*C
structure) side to join in.. maybe it would be allowed that you and I could process our own segments into the UTF-8 nodelist. Then and only
then would you need to "process" anything at your end.

One of the projects on my list is to get R16 going on the UTF-8 nodelist project as well. TTBOMK, nobody in R16 needs UTF-8 to properly spell their name, but in the interest of progress I'd like to be ready when/if that changes.

Andrew

--- GoldED+/LNX 1.1.5-b20160322
* Origin: Phoenix BBS * phoenix.bnbbbs.net (1:320/219)

Hello Andrew,

On 26 Nov 16 17:11, Andrew Leary wrote to Nicholas Boel:

One of the projects on my list is to get R16 going on the UTF-8
nodelist project as well. TTBOMK, nobody in R16 needs UTF-8 to
properly spell their name, but in the interest of progress I'd like to
be ready when/if that changes.

My last name is actually Boël. I haven't ever really had a chance to use it on computers until now. That's why I'm interested. I'll have to see if the rest of the chain above me would be interested (RJ Clay and Janis, basically). But if they're not, I'd gladly submit straight to Michiel myself.

Regards,
Nick

... "Не знаю. Я здесь только работаю."
--- GoldED+/LNX 1.1.5-b20160827
* Origin: thePharcyde_ distribution system (Wisconsin) (1:154/10)

Hello Andrew,

On Saturday November 26 2016 17:11, you wrote to Nicholas Boel:

One of the projects on my list is to get R16 going on the UTF-8
nodelist project as well.

Great!

TTBOMK, nobody in R16 needs UTF-8 to properly spell their name, but in
the interest of progress I'd like to be ready when/if that changes.

If you can not persuade Daniel Torrey to write his name as "Daniël", you can always change your system name to Phænix BBS for the occasion. Just to show the world that you can.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20130111
* Origin: Blijf Tønijn (2:280/5555)

Hello Nicholas,

On Saturday November 26 2016 21:57, you wrote to Andrew Leary:

My last name is actually Boël. I haven't ever really had a chance to
use it on computers until now.

You do not use it "on the Internet"? That should be easy, the web and almost all mail apllication are fully "internatinal".

That's why I'm interested. I'll have to see if the rest of the chain
above me would be interested (RJ Clay and Janis, basically). But if they're not, I'd gladly submit straight to Michiel myself.

"Submit directly to me" won't do much good. I am not the UTF master nodelist clerck. I may have been one of the driving forces, but as far as the production of the UTF nodelist is concerned I am just the maintainer of the R28 segment.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20130111
* Origin: Blijf Tønijn (2:280/5555)

Hello Michiel,

On 27 Nov 16 22:43, Michiel van der Vlist wrote to Nicholas Boel:

"Submit directly to me" won't do much good. I am not the UTF master nodelist clerck. I may have been one of the driving forces, but as far
as the production of the UTF nodelist is concerned I am just the maintainer of the R28 segment.

Ah yes, I stand corrected.

Regards,
Nick

... "Не знаю. Я здесь только работаю."
--- GoldED+/LNX 1.1.5-b20160827
* Origin: thePharcyde_ distribution system (Wisconsin) (1:154/10)