July 20, 2018

Cross Language 128 and 256 Bit AES Encryption

Filed under: Scripting,Web/Tech — Marcus Tettmar @ 10:47 am

With Macro Scheduler 14.4.10 we have added new 128 and 256 bit AES Encryption methods which interoperate well with other languages.

Cross-platform/cross-language encryption/decryption can be tricky. Different text encodings, different padding formats and understanding how different algorithms derive keys and initialisation vectors can make encrypting and decrypting between one language and another a bit of a pain!

Macro Scheduler’s AES function originally worked only on Unicode strings as this is the standard string format in Macro Scheduler.  Having had some issues trying to make this work between NodeJS and PHP we decided to create some new UTF8 based AES options (available in Macro Scheduler 14.4.10) which we have verified work well with PHP/OpenSSL and NodeJS.  

These new implementations provide 128 and 256 bit AES encryption.  They use CBC chaining method and use a SHA256 password. If using AES 128 the SHA256 password is truncated to the 32 byte key length.  By default the initialisation vector is set to “0000000000000000” but can be set to whatever you want (but must be 16 characters long). Padding is PKCS#5.

Here is an example of AES_256:

Let>AES_ALG=AES_256_CBC
AESEncrypt>hello world,mypassword,ENCRYPT,result
AESEncrypt>result,mypassword,DECRYPT,original


Here is a compatible PHP example:

data = "hello world";

$method = 'AES-256-CBC';

// simple password hash
$password = 'mypassword';
$key = hex2bin(substr(hash('sha256', $password),0,64));

echo "Method: " . $method . "\n";
$encrypted = encrypt($data, $key, $method);
echo "Encrypted: ". $encrypted . "\n";
$decrypted = decrypt($encrypted, $key, $method);
echo "Decrypted: ". $decrypted . "\n"; // plain text

function encrypt(string $data, string $key, string $method): string
{
  $iv = "0000000000000000";
  $encrypted = openssl_encrypt($data, $method, $key, OPENSSL_RAW_DATA, $iv);
  $encrypted = base64_encode($encrypted);
  return $encrypted;
}

function decrypt(string $data, string $key, string $method): string
{
  $data = base64_decode($data);
  $iv = "0000000000000000";
  $data = openssl_decrypt($data, $method, $key, OPENSSL_RAW_DATA,$iv);
  return $data;
}

Try this PHP Code Here

And here’s the Javascript equivalent:

let data = 'hello world';
let password = 'mypassword';
let iv = '0000000000000000';

let password_hash = crypto.createHash('sha256').update(password,'utf8').digest('hex');

let key = hex2bin(password_hash);
password_hash = Buffer.alloc(32,key,"binary");

let cipher = crypto.createCipheriv('aes-256-cbc', password_hash, iv);

let encryptedData = cipher.update(data, 'utf8', 'base64') + cipher.final('base64');

console.log('Base64 Encrypted:', encryptedData);

let decipher = crypto.createDecipheriv('aes-256-cbc', password_hash, iv);

let decryptedText = decipher.update(encryptedData, 'base64', 'utf8') + decipher.final('utf8');

console.log('Decrypted Text:', decryptedText)

function hex2bin(hex)
{
  var bytes = [], str;

  for(var i=0; i< hex.length-1; i+=2)
    bytes.push(parseInt(hex.substr(i, 2), 16));

  return String.fromCharCode.apply(String, bytes);
}

Try this Javascript code here.

You can try the Javscript and PHP examples at repl.it:

https://repl.it/@MarcusTettmar/phpaes256cbc1
https://repl.it/@MarcusTettmar/jsaes267cbc1

April 2, 2018

Escaping File Paths in HTTP Requests

Filed under: Automation,Web/Tech — Marcus Tettmar @ 5:48 pm

If you ever need to send a path as a URL parameter in an HTTP request, be sure to “escape” it to avoid the path delimiters (“\” symbols) being seen as delimiters in the URL path.

E.g. let’s say you need to send a filename as a parameter in a GET request, like this:

https://someserver.com/resource?filename=c:\my files\subfolder\filename.txt

If we fail to escape those “\” characters we are likely to get a server error or a 404 not found error at best. The space character in the above example is also likely to upset things!

We can solve this by using VBScript’s Escape function to “URL Encode” the string. What this does is replace the special characters with a special code made of a % symbol and two hexadecimal digits. E.g. a space character is replaced with %20.

So, we should use the Escape function something like this:

Let>strFilename=c:\my files\subfolder\filename.txt
VBEval>Escape("%strFilename%"),strFilename

Then we can safely do:

HTTPRequest>https://someserver.com/resource?filename=%strFilename%,,GET,,strResponse

If you were to paste the URL into Google Chrome or IE you will find that behind the scenes they apply this encoding for you.

Tip: A great way to test HTTP requests is to use http://webhook.site – this gives you a special URL you can send your data to (in place of your real web service) and for each request you make it will show you what it received. Try manually sending a filename like the one above using your web browser and you’ll see those % codes being added for you.

November 24, 2015

Finding HTML Attributes For Automating Web Sites

Filed under: Automation,Scripting,Web/Tech — Marcus Tettmar @ 1:00 pm

If you’re new to automating IE/websites with WebRecorder or the native Macro Scheduler IE functions you may be wondering how to determine which elements and attributes to use.

In this video I demonstrate how to use IE’s F12 key to invoke Developer Tools and use that to quickly find the elements we’re interested in and the attributes we need to use:

(You might want to click on the video toolbar to select a larger resolution size, view full screen or view on YouTube so that you can see the code).

January 24, 2012

WebRecorder 3.0 – Faster, More Reliable, More Functions

Filed under: Automation,Web/Tech — Marcus Tettmar @ 11:04 pm

We have today released WebRecorder 3.0.

WebRecorder 3.0 - Faster more reliable internet macros

This is a major rewrite with a better recording engine capable of recording more tags and producing leaner, cleaner, code and a faster and more reliable runtime.

It also introduces a download manager so that file downloads can be recorded and scripted more easily without recourse to sending keystrokes to IE’s standard file download box.

We’ve improved the existing commands so that you can script more tags and use more attributes and added new functions for retrieving data, retrieving the screen position of elements and interfacing with the new native IE functions in Macro Scheduler 13.1.

Oh, and you no longer need to use the mouse to record clicks and form fills!

  • Cleaner, leaner, faster code.
  • More reliable method for waiting for documents to complete loading
  • Download manager for scripting file downloads
  • Better error trapping
  • Ability to set timeout for Clicks and Form Fills etc
  • Click and Form fill functions will wait until target tags exist (within timeout)
  • New functions for extracting data
  • New function to get X,Y screen position of elements
  • No need to use the mouse to identify recorded elements
  • New, improved UI look and feel
  • Runtime component integrates better with Macro Scheduler’s script controller (faster termination, less liable to hang ups when aborting scripts etc)
  • Interface easily with Macro Scheduler’s IEGetTags and IETagEvent function
  • New script function for waiting for text in the page before continuing
  • Functions to automatically set extracted tag buffer size and avoid buffer overruns

More information on WebRecorder here.  WebRecorder 3.0 scripts require Macro Scheduler 13.1 to run.

Enterprise customers with in-date maintenance can already download WebRecorder 3.0 from the registered download area.

Other customers can view upgrade options in their registered download account, or download a trial version here.

Trial Downloads | Registered Downloads | Upgrades

May 20, 2011

Scraping Data From Web Pages

Filed under: Automation,Scripting,Web/Tech — Marcus Tettmar @ 1:02 pm

I’ve seen quite a lot of requests lately from people wanting to know how to extract text from web pages.

Macro Scheduler’s optional WebRecorder add-on simplifies the automation of web pages and includes functions for extracting tables, text or HTML from web page elements. WebRecorder’s Tag Extraction wizard makes it easy to create the code.

Sometimes you can choose a specific HTML element and identify it uniquely via it’s ID or NAME attribute. But other times you might want all the text from the whole page, or you may need to extract the entire page and then parse out the bits you’re interested in using RegEx or some other string manipulation functions.

To extract an entire page I specify the BODY element. If you want to extract data from web pages it does help if you know a little about HTML. And if you do you’ll know that each page has just one BODY element which contains the code making up the visible portion of the page.

Here’s code produced using WebRecorder when navigating to mjtnet.com and using the Tag Extraction wizard to extract the BODY text:

IE_Create>0,IE[0]

IE_Navigate>%IE[0]%,http://www.mjtnet.com/,r
IE_Wait>%IE[0]%,r
Wait>delay

//Modify buffer size if required (you may get a crash if buffer size too small for data) ...
Let>BODY0_SIZE=9999
IE_ExtractTag>%IE[0]%,,BODY,0,0,BODY0,r
MidStr>r_6,1,r,BODY0

MessageModal>BODY0

The macro simply displays just the text in a message box but could be set to pull out the full HTML. You could then parse it with RegEx to get the information you are interested in.

You will need WebRecorder installed for the above to work.

If you don’t have WebRecorder you can do the same with a bit more work using VBScript. Some library functions for doing this can be found here and here.

So here’s the equivalent in VBScript:

VBSTART
Dim IE

'Creates IE instance
Sub CreateIE
  Set IE = CreateObject("InternetExplorer.Application")
  IE.Visible=1
End Sub

'Navigate to an IE instance
Sub Navigate(URL)
  IE.Navigate URL
  do while IE.Busy
  loop
End Sub

'This function extracts text from a specific tag by name and index
'e.g. TABLE,0 (1st Table element) or P,1 (2nd Paragraph element)
'set all to 1 to extract all HTML, 0 for only inside text without HTML
Function ExtractTag(TagName,Num,all)
  dim t
  set t = IE.document.getElementsbyTagname(Tagname)
  if all=1 then
    ExtractTag = t.Item(Num).outerHTML
  else
    ExtractTag = t.Item(Num).innerText
  end if
End Function
VBEND

VBRun>CreateIE
VBRun>Navigate,www.mjtnet.com

VBEval>ExtractTag("BODY",0,0),BodyText
MessageModal>BodyText

But what if you already have a macro which already opens IE, or works against an already open instance of IE? The above macros need to create the IE instance before they can access them and extract data from them. You may have a macro that already starts IE some other way – maybe just by using a RunProgram or ExecuteFile call, or indirectly via some other application. Many times people tackle the extraction of data from such an IE window by sending keystrokes to do a Select-All, Edit/Copy and then use GetClipboard; or even File/Save As to save the HTML to a file. This of course adds time and can be unreliable. So how else can we do it?

Well, this tip shows us a function we can use to attach to an existing IE instance. So let’s use that and then use our ExtractTag function to pull out the BODY HTML:

VBSTART
Dim IE

' Attaches to an already running IE instance with given URL
Sub GetIE(URL)
  Dim objInstances, objIE
  Set objInstances = CreateObject("Shell.Application").windows
  If objInstances.Count > 0 Then '/// make sure we have instances open.
    For Each objIE In objInstances
      If InStr(objIE.LocationURL,URL) > 0 then
        Set IE = objIE
      End if
    Next
  End if
End Sub

'This function extracts text from a specific tag by name and index
'e.g. TABLE,0 (1st Table element) or P,1 (2nd Paragraph element)
'set all to 1 to extract all HTML, 0 for only inside text without HTML
Function ExtractTag(TagName,Num,all)
  dim t
  set t = IE.document.getElementsbyTagname(Tagname)
  if all=1 then
    ExtractTag = t.Item(Num).outerHTML
  else
    ExtractTag = t.Item(Num).innerText
  end if
End Function
VBEND

VBRun>GetIE,www.mjtnet.com

VBEval>ExtractTag("BODY",0,1),BodyHTML
MessageModal>BodyHTML

This snippet assumes a copy of IE is already open and pointing to www.mjtnet.com. The GetIE call creates a link to that IE window and then we use the ExtractTag function to pull out the HTML of the BODY element.

These examples use the BODY element, which will contain everything displayed on the page. As I mentioned before you can be more specific and specify some other element, and with WebRecorder, or a modified version of the ExtractTag VBScript function use other attributes to identify the element (the existing VBScript ExtractTag function shown above just uses the numeric index). WebRecorder tries to make it simple by giving you a point and click wizard, making some assumptions for you, so that you need not fully understand the HTML of the page. But it still helps you understand HTML. Looking at the source of the page you should be able to identify the element you need to extract from. And whether you extract directly from that or extract the BODY and then use RegEx being prepared to delve into the HTML source is going to get you further.

UPDATE: 19th January 2012

As of version 13.0.06 Macro Scheduler now includes a function called IEGetTags. For a given tag type and IE tab this will retrieve an array of tag contents. It can extract just the text, or html of the tags. This example extracts the inner HTML of all DIV elements in the open IE document currently at www.mjtnet.com:

IEGetTags>mjtnet.com,DIV,H,divArr

You can then cycle through each one with a Repeat Until

If>divArr_count>0
  Let>k=0
  Repeat>k
    Let>k=k+1
    Let>this_div_html=divArr_%k%
    .. 
    .. do something with it
    .. e.g. use RegEx or substring searching to determine 
    .. if this is the DIV you want and extract from it
    .. 
  Until>k=divArr_count
Endif

To further identify the tag you are interested in, or find the data you want, you can use RegEx, EasyPatterns, or string functions.

Macro Scheduler 13.0.06 and above also has a function called IETagEvent which will let you simulate a Click on a given tag, focus it, or modify its value. So once you have identified a tag using IEGetTags and your Repeat/Until loop you can click on it, focus it or modify its value (e.g. for form fields).

May 25, 2010

Tweetlib: A DLL Plugin for Tweeting Status Updates via oAuth

Filed under: Announcements,Automation,Scripting,Web/Tech — Marcus Tettmar @ 4:23 pm

As noted yesterday I have been waiting on Twitter to provide xAuth access. They declined, saying it was not appropriate. I’m not really sure why.

No matter, I decided to make a small DLL to simplify Tweeting from Macro Scheduler. It uses the full oAuth interface.

Implementing oAuth in Macro Scheduler code would require lots of VBScript code and would be very complicated (although doable in theory). So instead I decided to create a DLL which you can use in Macro Scheduler to tweet in one line of code.

You can download it here.

And then to post a status update all you need to do is something like this:

Let>message=Hello from Macro Scheduler
LibFunc>%SCRIPT_DIR%\tweetlib.dll,UpdateStatus,r,message,buff,1024

Note that the first time you call UpdateStatus you will be asked to log into Twitter and click “Allow” to authorise Macro Scheduler to access your account. You will then be given a PIN to enter. You only need to do this once. If you ever need to revoke access and start over call the RemoveCredentials function. Your Twitter username and password are NOT stored anywhere. This uses the oAuth authorisation scheme which provides an access token. It is the access token which is stored and this only allows Macro Scheduler to access the API for your account.

The return buffer will contain the XML of the status update operation if successful or an error message if not.

See readme.txt and sample .scp in the zip file. Enjoy.

May 24, 2010

Tweeting from Macro Scheduler Without the API

Filed under: Automation,Scripting,Web/Tech — Marcus Tettmar @ 1:51 pm

A while back I posted an article showing how to Tweet via Twitter‘s API. It uses basic authentication which Twitter plan to turn off in the near future. The alternative, oAuth is awkward for desktop based apps, but xAuth is now available and should be doable in Macro Scheduler. I have requested xAuth access from Twitter and, assuming it’s doable, will try and provide an example once I’ve received it and tried it out.

In the mean time it occurred to me that we don’t really need an API if all we want to do is send a status update. We can do that easily using Macro Scheduler and WebRecorder functions by controlling an instance of Internet Explorer, which can be done in the background.

Below is a script which demonstrates this. It offers a function called LoginToTwitter which need only be called once per session, and an UpdateStatus function to update your status. Just set your Twitter username and password in the first two lines and you should be all set.

Let>TW_USERNAME=XXXXX
Let>TW_PASSWORD=XXXXX

//only need do this once per session
GoSub>LoginToTwitter

GoSub>UpdateStatus,This is a test

GoSub>UpdateStatus,This is a test 2

GoSub>LogOut

// END

//*** SUBROUTINES ***//
SRT>LoadWR
  //load the WebRecorder runtime
  LibLoad>IEAuto.dll,hIE
  If>hIE=0
    MessageModal>Could not load IEAuto.dll, make sure it is in the path or edit the LibLoad line.
    Exit>0
  EndIf
END>LoadWR

SRT>LoginToTwitter
  GoSub>LoadWR
  //open IE
  LibFunc>hIE,CreateIE,ieTwitter,0
  LibFunc>hIE,ShowIE,res,ieTwitter,1

  //log in to Twitter
  LibFunc>hIE,Navigate,r,ieTwitter,http://twitter.com/login
  LibFunc>hIE,WaitIE,r,ieTwitter
  LibFunc>hIE,FormFill,r,ieTwitter,,,session[username_or_email],TW_USERNAME,0
  LibFunc>hIE,FormFill,r,ieTwitter,,,session[password],TW_PASSWORD,submit

  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
END>LoginToTwitter

SRT>UpdateStatus
  LibFunc>hIE,FormFill,r,ieTwitter,,,status,UpdateStatus_VAR_1,submit
  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
END>UpdateStatus

SRT>LogOut
  LibFunc>hIE,KillIE,r,ieTwitter
END>LogOut

You need the IEAuto.DLL library which is installed with WebRecorder.

For a bit of fun the following code copies the currently highlighted text to the clipboard and tweets it. So assigned to a hot key it can be used to tweet any text from any application.

Press CTRL
Send>c
Release CTRL
WaitClipBoard
GetClipBoard>theText
GoSub>UpdateStatus,theText

The real challenge is finding something useful to do with it! 🙂

March 4, 2010

Choose Your Web Browser

Filed under: General,Web/Tech — Marcus Tettmar @ 10:39 am

One of my Windows XP systems installed an automatic update last night when I shut it down.  This morning I was presented with this:

Windows EU WebBrowser Choice

As I understand it this is as a result of an EU competition law decision.   Is this a good thing or a bad thing?

On the one hand it seems a waste of money and illogical.

I’m no fan of IE, but why shouldn’t Microsoft build a web browser and make it part of their operating system?  Should the EU force them to offer a choice of text editors, calculators, paint programs and calendar applications too? Where should it end?

Plus, yet another dialog like this is surely going to confuse the ordinary user.  Many have no idea what a “web browser” is and just call it “My Internet” or “Google”.  The dialog preceding the one above has to go to the trouble of explaining what a web browser is.  And isn’t the user just going to take the first option anyway?  In which case should the EU force Microsoft to make IE8 something other than the default option?

But then if I were trying to sell a web browser I guess I would welcome this decision. It might give me a little extra exposure.

What do you think?

January 29, 2010

Running Macro Scheduler Macros over the Web Via PHP

Filed under: Automation,General,Scripting,Web/Tech — Marcus Tettmar @ 9:31 am

Here’s a quick and simple proof of concept for running Macro Scheduler macros via the web and having their output displayed in the user’s browser:

Running Macro Scheduler Macros Over the Web Via PHP

The screenshot shows the PHP script, Macro Scheduler script and Internet Explorer being used to run the macro.

1) If you don’t already have a Windows based web server with PHP running, download and install WAMPServer. It’s easy.

2) Create a simple PHP script which takes an EXE name as a parameter and any parameters you want to pass to it. The following script will run EXEs that are in the c:\wamp\ folder, passing in any parameters provided and will dispay the EXEs output.

<?php
$exe = $_GET['exe'];

$dir = "c:\\wamp\\";

$parms = "";
foreach($_GET as $key=>$val) {
  $parms .= "/$key=$val ";
}

echo shell_exec("\"$dir$exe\" $parms");
?>

3) PHP’s safemode must be disabled for this script to work.

4) Create a Macro Scheduler macro and use SOWrite or SOWriteLn to output information. Compile it with the “Create Console App” option checked. Compile the EXE (or copy it) to the c:\wamp\ folder.

5) Now the macro can be executed via the web using http://servername/runmacro.php?exe=my.exe&parm1=value&etc=… which could be a link or entered into the browser directly.

6) Consider adding further security to the script to prevent anyone running any EXE on your server, or putting it in a password protected folder. I’ll leave that to you.

Macro Scheduler Enterprise comes with the msNet Remote Controller which includes a CGI module for running Macro Scheduler macros via web servers.

April 20, 2009

Twittering from Macro Scheduler with the Twitter API

Filed under: Scripting,Web/Tech — Marcus Tettmar @ 7:42 am

Way back in the deep and distant past when the Internet was new and Bill Gates thought it was just a passing fad, I remember reading about a Cola vending machine on a University campus that some frivolous young boffins hooked up to the ‘net so that you could check its inventory from anywhere in the world using an old fashioned network command called “finger”. Why? Because they could.

Fast forward to the technologies of the current day and the latest trend of Twitter, and history is repeating itself. In the last week I’ve read about a restaurant that can take orders via Twitter, a bakery tweeting the emergence of fresh loaves from the oven; and, utterly pointless, some guys who created a system which sends a tweet every time their cat enters or exits its cat flap. Why? Well, because they can I guess.

Not wanting to be left out I decided to write some Macro Scheduler code to tweet status updates and monitor replies. Why? Well there might be a good reason for being able to do this – I’m sure someone will have one. Perhaps you have a client who wants you to set up a system to monitor the movement of his cat, process restaurant orders, or your local baker wants an automated fresh-loaf tweeter! But mostly, it’s because we can.

You’ll find the Twitter API documentation here. Here’s the code to Tweet a status update:

Let>username=YOURTWITTERNAME
Let>password=YOURPASSWORD

//Tweet from Macro Scheduler
Let>url=http://%username%:%password%@twitter.com/statuses/update.xml
Let>message=Kitty just left the buildng
HTTPRequest>url,,POST,status=%message%,result

Being serious for a moment I can see how a macro that monitors an application might want to post status updates to Twitter, or a backup script could alert you by Twitter when there’s a problem. It might be a public system, but don’t forget that Twitter profiles can be made private too, and Tweets can be viewed on and sent from your BlackBerry, iPhone, or even by SMS.

The following script sets up a loop which monitors your Twitter stream for “mentions” of your username. This might form the basis of a script which retrieves orders. Perhaps it could listen to Twitter for commands and carry out actions based on what message was sent. Or perhaps you just want a macro which does something when a cat decides to head out for the night. Use your imagination.

Let>username=YOURTWITTERNAME
Let>password=YOURPASSWORD
Let>ini_file=%SCRIPT_DIR%\twit.ini
Let>_delay=30

VBSTART
VBEND

//monitor twitter username "mentions" loop
Label>monitor_loop

Let>url=http://%username%:%password%@twitter.com/statuses/mentions.xml
HTTPRequest>url,,GET,,result

//remove the  portion (I don't need it and it avoids distinguishing the text IDs from the user IDs.
RegEx>[^>](.*?),result,0,user_matches,nf,1,,result

//extract all texts
RegEx>(?<=)[^>]*?(?=),result,0,text_matches,num_texts,0
If>num_texts>0
  //extract all ids
  RegEx>(?<=)[^>]*?(?=),result,0,id_matches,num_ids,0

  //get last known
  Let>last_known_id=0
  IfFileExists>ini_file
    ReadIniFile>ini_file,SETTINGS,LAST_ID,last_known_id
  Else
    WriteLn>ini_file,wlnr,
  Endif

  //iterate through texts
  Let>k=0
  Repeat>k
    Let>k=k+1
    Let>this_id=id_matches_%k%
    If>this_id>last_known_id
      Let>msg_text=text_matches_%k%
      /*
      msg_text contains the message 
      Use your imagination here!
      For now we'll show it in a message
      */
      MessageModal>msg_text
    Endif
  Until>k=num_texts

  //store last ID
  EditIniFile>ini_file,SETTINGS,LAST_ID,id_matches_1
Endif

Wait>_delay
Goto>monitor_loop

The script retrieves the 20 most recent “mentions”. It stores the last seen ID in an INI file so that on the next check it ignores those it has seen before, only retrieving messages with a larger ID number.

This is a quick and dirty solution with no error checking, using RegEx to parse the XML that is returned by the call to Twitter. You may prefer to use the MS XML object as shown here.

Whether this proves useful or completely pointless, I hope you have fun. If you’re using Macro Scheduler with Twitter, please add a comment below to let us know how … and why!

Don’t forget you can follow me on Twitter where I may occassionally say something useful.

Older Posts »