UTF-8 edit ini file

Technical support and scripting issues

Moderators: Dorian (MJT support), JRL

Post Reply
User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

UTF-8 edit ini file

Post by Grovkillen » Tue Apr 07, 2020 7:56 am

If I create a ini file with UTF-8 activated the edit ini file command aren't complying. See this test:

Code: Select all

WriteLn>%SCRIPT_DIR%\inifile1.ini,,######
EditIniFile>%SCRIPT_DIR%\inifile1.ini,section,entry,åäöÅÄÖ

Let>WLN_ENCODING=UTF8
WriteLn>%SCRIPT_DIR%\inifile2.ini,,######
EditIniFile>%SCRIPT_DIR%\inifile2.ini,section,entry,åäöÅÄÖ

ExecuteFile>%SCRIPT_DIR%\inifile1.ini
ExecuteFile>%SCRIPT_DIR%\inifile2.ini
Let>ME=%Script%

Running: 15.0.24
version history

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: UTF-8 edit ini file

Post by Marcus Tettmar » Tue Apr 07, 2020 10:48 am

Hi,

Looks like EditIniFile isn't UTF-8/Unicode compatible. Have added to list.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: UTF-8 edit ini file

Post by Grovkillen » Tue Apr 07, 2020 10:55 am

Marcus Tettmar wrote:
Tue Apr 07, 2020 10:48 am
Hi,

Looks like EditIniFile isn't UTF-8/Unicode compatible. Have added to list.
Thanks :)
Let>ME=%Script%

Running: 15.0.24
version history

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: UTF-8 edit ini file

Post by Grovkillen » Sat May 02, 2020 8:48 am

FYI: the read ini file command isn't parsing UTF-8 as well.
Let>ME=%Script%

Running: 15.0.24
version history

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: UTF-8 edit ini file

Post by Grovkillen » Mon May 25, 2020 9:46 am

To devs: is this on the horizon (as in a fix)? Or should I create a separate "clean up script" for this? I would rather it be the first options since I'm relying on the ini files to only be updated if there's a difference.. this could trigger false positives since I cannot know for sure if the value that differ is in fact a correct (cleaned) one.
Let>ME=%Script%

Running: 15.0.24
version history

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: UTF-8 edit ini file

Post by Marcus Tettmar » Tue May 26, 2020 9:58 am

Not on the immediate horizon. Have looked into it and the hard fact of the matter is that Windows doesn't natively support utf8 ini files either. We're wrapping the Windows Api functions which read/write to INI files. They don't do it, so therefore nor do we. We would therefore need to reinvent the wheel from scratch and write our own INI file handling functions. Microsoft may argue that INI files are old tech and that use of the registry supersedes them. I'll need to look into this more and weigh up the effort against the need.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: UTF-8 edit ini file

Post by Grovkillen » Tue May 26, 2020 10:54 am

Oh, please don't be like Microsoft then ;) I love my INI files. They are so portable and easy to backup compared to the registry.
Let>ME=%Script%

Running: 15.0.24
version history

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: UTF-8 edit ini file

Post by Grovkillen » Sat May 30, 2020 2:49 am

This is how I parse a ini file in JS. Perhaps something that could be done with MS?

Code: Select all

    'iniFileToObject': function(string, pipe2array = false) {
      let object = {};
      let sections = string.match(/^\[[^\]\r\n]+](?:[\r\n]([^[\r\n].*)?)*/gm);
      for (let i=0; i < sections.length; i++) {
          let sectionName = sections[i].split("\n")[0];
          sectionName = sectionName.replace(/[\[\]]/g, '').trim();
          object[sectionName] = {};
          let key = sections[i].match(/^((?!\/\*[\s\S]*?\*\/|([^\\:]|^)\/\/.*$).)+/gm);  //we remove comments "//" syntax, single row
          for (let k=1; k < key.length; k++) {
              let keyValue = key[k].split("=");
              if ((keyValue[1].charAt(0) === "0" && keyValue[1].length > 1) || isNaN(Number(keyValue[1]))) {   // leading zeros are interpreted as string values
                  if (pipe2array && keyValue[1].trim().includes("|")) {
                      object[sectionName][keyValue[0].trim()] = keyValue[1].trim().split("|");
                  } else if (keyValue[1].trim() === "null") {
                      object[sectionName][keyValue[0].trim()] = null;
                  } else {
                      object[sectionName][keyValue[0].trim()] = keyValue[1].trim();
                  }
              } else {
                  object[sectionName][keyValue[0].trim()] = Number(keyValue[1]);
              }
          }
      }
      return object;
    }
I have added a "pipe2array" thing to be able to have some keys automatically turned into arrays by separating them with a | "pipe".

This parser allow for comments on single rows or on key value rows using "//". Section rows must not contain any comments.

Writing to a ini file would be just to check if a section and key is already present, if so replace the string of that key with the new value, remember the comment if any and add it to the end of the string.

If no section and/or key is found, inject a new line with this information.

For easier parsing you may even allow for the ini file comments to be part of the ini object as simply add them as such:

SECTION1=section1s_name
SECTION1_ROW=row_of_section_in_ini_file
SECTION1_KEY1=keyname
SECTION1_KEY1_ROW=row_of_key_in_ini_file
SECTION1_KEY1_VALUE=value_of_key1
SECTION1_KEY1_COMMENT=comment_found_after_value
...
SECTIONn=section1s_name
SECTIONn_ROW=row_of_section_in_ini_file
SECTIONn_KEYn=keyname
SECTIONn_KEYn_ROW=row_of_key_in_ini_file
SECTIONn_KEYn_VALUE=value_of_keyN
SECTIONn_KEYn_COMMENT=comment_found_after_value
+
INI_FILE_COMMENT1=comment_string
INI_FILE_COMMENT1_ROW=row_of_comment_in_ini_file
...
INI_FILE_COMMENTn=comment_string
INI_FILE_COMMENTn_ROW=row_of_comment_in_ini_file

Just as an idea of how you could implement it, I'm not suggesting you should allow for the user to see this information.

You could then say that a ini file syntax must be used as such:

Sections [section] must reside on a single row with no comments, spaces before and after a section entry will be trimmed away.
Key-value entries must reside on a single row, spaces before and after a key will be trimmed away.
Key must be followed by a = or else the row will be treated as a comment.
Values must be single row and everything after a "//" is treated as a comment.

This way you wouldn't reinvent the wheel, simply make a better propulsion mechanism. And I would love MS even more :)
Let>ME=%Script%

Running: 15.0.24
version history

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: UTF-8 edit ini file

Post by Grovkillen » Sun Jun 07, 2020 2:17 pm

Okay, I really hit rock bottom with this problem. I want to read the INI file and lookup a folder structure based on info in the INI. The illegal characters will not work when parsing the path to the file based on text with none-UTF-8 type. :cry:
Let>ME=%Script%

Running: 15.0.24
version history

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: UTF-8 edit ini file

Post by Grovkillen » Sun Jun 07, 2020 5:06 pm

Okay, I managed to get my server side stuff to work by moving the encoding to the front-end. I read the ini files and parse them as binary data. So I'm all right with that solution. So no immediate hurry for my sake :)

I.E. I now use the default encoding (ANSI) and the non-ANSI characters are converted in the front-end instead of on server side script.
Let>ME=%Script%

Running: 15.0.24
version history

hagchr
Automation Wizard
Posts: 327
Joined: Mon Jul 05, 2010 7:53 am
Location: Stockholm, Sweden

Re: UTF-8 edit ini file

Post by hagchr » Sun Jun 07, 2020 5:46 pm

Good you have a working solution. Not sure if relevant/helps, I played around and note that if I create and save an empty inifile2.ini in Notepad (as UTF-8), then when running your script in MS it looks ok (shows åäö).

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: UTF-8 edit ini file

Post by Grovkillen » Sun Jun 07, 2020 6:34 pm

hagchr wrote:
Sun Jun 07, 2020 5:46 pm
Good you have a working solution. Not sure if relevant/helps, I played around and note that if I create and save an empty inifile2.ini in Notepad (as UTF-8), then when running your script in MS it looks ok (shows åäö).
Thanks for testing. I didn't do that test but now I learned how to parse binary data in JavaScript. That's something positive :D
Let>ME=%Script%

Running: 15.0.24
version history

Post Reply
cron
Sign up to our newsletter for free automation tips, tricks & discounts