July 20, 2018

Cross Language 128 and 256 Bit AES Encryption

Filed under: Scripting,Web/Tech — Marcus Tettmar @ 10:47 am

With Macro Scheduler 14.4.10 we have added new 128 and 256 bit AES Encryption methods which interoperate well with other languages.

Cross-platform/cross-language encryption/decryption can be tricky. Different text encodings, different padding formats and understanding how different algorithms derive keys and initialisation vectors can make encrypting and decrypting between one language and another a bit of a pain!

Macro Scheduler’s AES function originally worked only on Unicode strings as this is the standard string format in Macro Scheduler.  Having had some issues trying to make this work between NodeJS and PHP we decided to create some new UTF8 based AES options (available in Macro Scheduler 14.4.10) which we have verified work well with PHP/OpenSSL and NodeJS.  

These new implementations provide 128 and 256 bit AES encryption.  They use CBC chaining method and use a SHA256 password. If using AES 128 the SHA256 password is truncated to the 32 byte key length.  By default the initialisation vector is set to “0000000000000000” but can be set to whatever you want (but must be 16 characters long). Padding is PKCS#5.

Here is an example of AES_256:

Let>AES_ALG=AES_256_CBC
AESEncrypt>hello world,mypassword,ENCRYPT,result
AESEncrypt>result,mypassword,DECRYPT,original


Here is a compatible PHP example:

data = "hello world";

$method = 'AES-256-CBC';

// simple password hash
$password = 'mypassword';
$key = hex2bin(substr(hash('sha256', $password),0,64));

echo "Method: " . $method . "\n";
$encrypted = encrypt($data, $key, $method);
echo "Encrypted: ". $encrypted . "\n";
$decrypted = decrypt($encrypted, $key, $method);
echo "Decrypted: ". $decrypted . "\n"; // plain text

function encrypt(string $data, string $key, string $method): string
{
  $iv = "0000000000000000";
  $encrypted = openssl_encrypt($data, $method, $key, OPENSSL_RAW_DATA, $iv);
  $encrypted = base64_encode($encrypted);
  return $encrypted;
}

function decrypt(string $data, string $key, string $method): string
{
  $data = base64_decode($data);
  $iv = "0000000000000000";
  $data = openssl_decrypt($data, $method, $key, OPENSSL_RAW_DATA,$iv);
  return $data;
}

Try this PHP Code Here

And here’s the Javascript equivalent:

let data = 'hello world';
let password = 'mypassword';
let iv = '0000000000000000';

let password_hash = crypto.createHash('sha256').update(password,'utf8').digest('hex');

let key = hex2bin(password_hash);
password_hash = Buffer.alloc(32,key,"binary");

let cipher = crypto.createCipheriv('aes-256-cbc', password_hash, iv);

let encryptedData = cipher.update(data, 'utf8', 'base64') + cipher.final('base64');

console.log('Base64 Encrypted:', encryptedData);

let decipher = crypto.createDecipheriv('aes-256-cbc', password_hash, iv);

let decryptedText = decipher.update(encryptedData, 'base64', 'utf8') + decipher.final('utf8');

console.log('Decrypted Text:', decryptedText)

function hex2bin(hex)
{
  var bytes = [], str;

  for(var i=0; i< hex.length-1; i+=2)
    bytes.push(parseInt(hex.substr(i, 2), 16));

  return String.fromCharCode.apply(String, bytes);
}

Try this Javascript code here.

You can try the Javscript and PHP examples at repl.it:

https://repl.it/@MarcusTettmar/phpaes256cbc1
https://repl.it/@MarcusTettmar/jsaes267cbc1

May 15, 2018

Nice Customer Testimonial

Filed under: General — Marcus Tettmar @ 12:20 pm

Just had this testimonial from a recent customer and had to share it with you:

“Just wanted to take some time to say thank you for helping us out on this project at such short notice. We’d sunk quite some time into getting up to speed with a competing product, and had bought personal licenses as well as evaluation and then to find that it could not script all the parts we needed, and frankly how terrible their support was left us in a very bad spot – thank God we’d not stumped up the £3k for their .NET module which we were on the verge of doing. We quickly ran through 4 other web automation packages – none of which were up to the job for our task, or the learning curve was too steep to find out – then stumbled on your Macro Scheduler. If only we’d got to you first. Thanks again, I have no doubt we’ll find use for Macro Scheduler on other projects – it’s a very useful tool to have in the tool-box now.”

You can read more testimonials here.

May 4, 2018

Macro Scheduler Discounts for Students, Charities and Not-for-Profits

Filed under: Announcements,General — Marcus Tettmar @ 1:21 pm

Students

If you are a student and want to use Macro Scheduler for your studies, please contact us with details and if everything checks out we will give you a 40% discount coupon.

Charities and Not-for-profits

Charities and not-for-profits do amazing things around the world. From small local groups feeding the poor to large organizations addressing global issues, these entities do incredible work often on a shoe string budget. We here at MJT Net Ltd admire the work these people do and hope that by offering them a 40% discount on our software we can support them.

If you represent a charity or not-for-profit organisation please contact us with your details for a 40% discount coupoon.

All we ask in return is that you tell all the people you work with about our software, and how it can help them become more productive (and therefore save your organisation even more money!)

So that this offer is used fairly we ask that you declare that you meet the following requirements:

  • You must have legal charitable or not-for-profit status in your country
  • Must direct at least 90% of any funding or donations to the actual work, and not administration
  • Must have no more than US$2 million in government funding per year
  • Must have a website with your mission and clear outcomes from your work
  • Your organisation must not discriminate against any particular race, age, religion or sexual persuasion

As an alternative to purchasing a license and ongoing maintenance you may wish to consider a monthly subscription.

April 19, 2018

Macro Scheduler Subscriptions from only 12.50/month

Filed under: Announcements,General — Marcus Tettmar @ 8:40 am

Did you know we offer monthly, quarterly or yearly subscriptions for Macro Scheduler Standard?

These subscriptions have no minimum term, so can work out very cost effective if you only need the product for a short while. You may also prefer the budgeting aspect of fixed priced subscriptions over the up front cost of a perpetual license and then potential future upgrade or maintenance costs.  I know I do.

A subscription gives you access to the latest version of Macro Scheduler for as long as your subscription is active. So maintenance and upgrades are included and you never go out of date.

Check our the subscription options here.

 

April 5, 2018

Macro Scheduler 14.4.08 Available

Filed under: Announcements — Marcus Tettmar @ 4:42 pm

Macro Scheduler maintenance build 14.4.08 is now available for download from the usual places:

Registered Downloads/Upgrades | Trial Versions | New Licenses | Version History

This version introduces a simple and useful new WebRecorder function called IEGetAllText. It simply gets ALL the text in the specified Internet Explorer browser window, and it will iterate through all frames in the page too. A super simple way of scraping a web page or web app. You can tell it to get just text or html. Once you have that you could parse it with e.g. Regular Expressions.

While we already have functions that let you target a specific element based on a known attribute, this function gives you a quick way of pulling out all the text in the document.

April 4, 2018

Running Powershell Code From Inside Your Macros

Filed under: Automation,Scripting — Marcus Tettmar @ 4:25 pm

A while back I wrote this blog post which shows you how you can have a PowerShell script run Macro Scheduler commands via the MacroScript SDK.

However, what if you want to do it the other way around and run PowerShell code from inside a Macro Scheduler macro instead?

One way would simply be to run a PowerShell script on the command line, like this:

Run>powershell.exe -file "c:\myfolder\myscript.ps1"

But what if you didn’t have a PowerShell script file?  Well, while you could always have your macro create one using the WriteLn command (and delete it after), you don’t need to as you can pass a sequence of commands to PowerShell on the command line:

powershell.exe -executionPolicy bypass -command "Write-Host 'Hello World'; exit 123;"

As you can see all we’re really doing is passing the content of the script instead of a file.

We can also return the exit code of the PowerShell script to a variable in your macro.  Here’s an example:

//This line ensures the powershell script finishes before the macro continues
Let>RP_WAIT=1
//Uncomment this line to HIDE the powershell window when ready to go live
//Let>RP_WINDOWMODE=0

Let>localVar=Hello World
LabelToVar>MyPowershellScript,theScript
Run>powershell.exe -executionPolicy bypass -command "%theScript%"

//display the exit code
MDL>RP_RESULT

/*
MyPowershellScript:

$var = '%localVar%';
Write-Host $var;
Start-Sleep -s 10;
exit 555;
*/

What we’ve done here is store our PowerShell code in a label. We get that code into a variable and then pass it to PowerShell on the command line.

Notice how we can embed Macro Scheduler variables in the code too. We use PowerShell’s exit statement to set a return value which Macro Scheduler gets in the RP_RESULT variable.

Setting RP_WAIT to 1 makes sure the macro waits for PowerShell to finish before proceeding with the rest of the script. You may also want to hide the PowerShell window by setting RP_WINDOWMODE to 0, but for testing purposes I’ve left that commented out so that you can actually see it working. That’s also the reason for the 10 second delay at the end of the PowerShell script – apart from showing how we can add multiple lines of code it also gives us a chance to see things happen!

Get Date of Last Monday

Filed under: Scripting — Marcus Tettmar @ 2:59 pm

Someone asked me the other day how to get the date of the most recent Monday.  I came up with two solutions.

Get Date of Last Monday with VBScript

This version works by looping the date backwards one day at a time until the weekday of that date is a Monday.

//Put this VBSTART ... VBEND block at the top of your code ONCE
VBSTART
  Function LastMonday
    LastMonday = Date()
    Do While WeekDay(LastMonday) <> 2
      LastMonday = LastMonday -1
    Loop
  End Function
VBEND

//use this line whenever you want to get the date of last monday
VBEval>LastMonday,dateOfLastMonday

To make it work for a different day just change the number it compares the WeekDay to in the Do line. 1 is Sunday, 2 is Monday and so on.

Get Date of Last Monday with MacroScript

This version works out how many days we need to deduct from the current date to get to Monday based on what the current day of the week is. If it’s a Sunday we just deduct 6. If it’s a Monday or onward we deduct the current day number.

//DayOfWeek starts with 1 as Sunday, so Monday is 2.
If>curDayOfWeek>1
  //today must be Monday or later ... 
  //So diff we want is 2 - current day of week
  Let>diffToMonday=2-curDayOfWeek
  //e.g. if today Monday result would be 0 difference, Tuesday would yield -1 difference, Wednesday, -3 etc
Else
  //curDayOfWeek is 1, so ...
  //today must be sunday, so we just want 6 days ago
  Let>diffToMonday=-6
Endif

//This should give us number of days to subtract from current day 
//to get Monday's date
DateAdd>today,D,%diffToMonday%,mbeg

April 2, 2018

Escaping File Paths in HTTP Requests

Filed under: Automation,Web/Tech — Marcus Tettmar @ 5:48 pm

If you ever need to send a path as a URL parameter in an HTTP request, be sure to “escape” it to avoid the path delimiters (“\” symbols) being seen as delimiters in the URL path.

E.g. let’s say you need to send a filename as a parameter in a GET request, like this:

https://someserver.com/resource?filename=c:\my files\subfolder\filename.txt

If we fail to escape those “\” characters we are likely to get a server error or a 404 not found error at best. The space character in the above example is also likely to upset things!

We can solve this by using VBScript’s Escape function to “URL Encode” the string. What this does is replace the special characters with a special code made of a % symbol and two hexadecimal digits. E.g. a space character is replaced with %20.

So, we should use the Escape function something like this:

Let>strFilename=c:\my files\subfolder\filename.txt
VBEval>Escape("%strFilename%"),strFilename

Then we can safely do:

HTTPRequest>https://someserver.com/resource?filename=%strFilename%,,GET,,strResponse

If you were to paste the URL into Google Chrome or IE you will find that behind the scenes they apply this encoding for you.

Tip: A great way to test HTTP requests is to use http://webhook.site – this gives you a special URL you can send your data to (in place of your real web service) and for each request you make it will show you what it received. Try manually sending a filename like the one above using your web browser and you’ll see those % codes being added for you.

March 28, 2018

Will robotic process automation take your job away?

Filed under: Automation,General,Macro Recorder — Marcus Tettmar @ 2:25 pm

TLDR: No! That’s not what we’re here for. Robotic process automation is here to help you.

We often get called in to help a company automate a process. Many times this involves automating something that an end-user does. To make macros that make their job easier, less painful, and make them more productive.

Initial Resistance to Change

Sometimes, at the outset, we (or the IT department proposing Macro Scheduler as a solution) detect a little resistance from the end user. If not properly briefed they may be concerned that what we want to do is take work away from them, and they may be concerned that the macros, or software robots, are about to take their jobs away.

Epiphany!

Nothing could be further from the truth and by the time we’re done, the end users love us to bits and usually give us a long list of other things they’d like us to automate! Often they discover how easy it is to do it themselves with tools like the Macro Recorder and other code wizards.

I’ve written thousands of macros for hundreds of companies. I’ve never worked on anything that was designed to take a job away from an employee, other than one the employee didn’t like doing and didn’t have time to do without it impacting on their core purpose. In every case the macros we wrote were there to make the employees’ jobs easier, or to allow them to be more productive in more important areas. For example: removing a tedious admin task from an already overstretched clinician, allowing them to see more patients.

I suppose you could argue that the many over-night data-entry style macros we’ve written could have been done instead by a team of data entry personnel. But such a team didn’t exist to start with, or would have been cost prohibitive, or too error-prone to be considered in the first place.

Improving Productivity

So to answer the question, no, I don’t believe robotic process automation is about taking jobs away. It’s not about replacing staff. In my 21 years of automating user interfaces this isn’t what I’ve seen. Instead, I’ve been rewarded with making people’s lives easier, removing cumbersome, painful, processes, or using macros to simplify and improve an existing process.

March 23, 2018

Screen Scraping with Macro Scheduler [Update]

Filed under: Automation — Marcus Tettmar @ 9:55 pm

What is Screen Scraping?

Screen Scraping is a term used to describe the process of a computer program or macro extracting data from the display output of another application. Rather than parsing data from the database or data files belonging to an application, Screen Scraping pulls the data from the screen itself, extracting data that was intended to be displayed to the end-user as opposed to data designed for output to another application or database.  

Screen Scraping is necessary when there is a need to access the information displayed by the application but there is no method provided to access it behind the scenes. The database or data files may not be accessible, or may be undocumented or proprietary and therefore cannot be parsed easily; the costs associated with interacting with the database may be too high; or the license agreement or warranty prohibits it. In the case of legacy systems that are no longer supported there may be no knowledge of the data structures, or the technology used is no longer compatible with current technology. In these cases we are resorted to extracting the data from the screen – from the windows of the application.

The term Screen Scraping probably originates from the era of computer terminals when you could connect the terminal output of a computer to an input port on another and therefore record the screen data.  

Screen Scraping Methods

There are a number of ways we can retrieve information from the screen using Macro Scheduler, depending on the type of application the data is in.

Screen Scraping Web Applications

Applications like Macro Scheduler’s WebRecorder can access the data and objects inside an Interner Explorer window and can therefore be used to extract the data.  Technically speaking I would not call this screen scraping since WebRecorder is using an API interface provided by Internet Explorer, but the process of extracting information from web sites is commonly refered to as Screen Scraping.  With WebRecorder we can use the ExtractTag wizard to create code that extracts the text from a particular element in the page.   While WebRecorder is the easiest way to do it, it is also possible to automate IE and extract data from web pages by using VBScript. The following forum posts may help:

Automate Internet Explorer with OLE/ActiveX
Automate web forms with IE
HTTP GET and POST using VBscript

Screen Scraping Microsoft Office Applications

Microsoft Office Applications, like Internet Explorer, have a COM interface that allows scripts to manipulate them and access the data held within them.  Again, not really scraping data from the screen itself, as you are getting it directly from a programming interface. There are a number of examples in the forums and blog archives and also some sample scripts that come with Macro Scheduler which demonstrate how to automate Office applications and retrieve data from them.  

Working with Excel

Screen Scraping Regular Windows Applications

Most other applications don’t offer a scripting interface like MS Office or Internet Explorer.  This is where we really need to work directly with the screen.   There are a number of ways we can do this kind of Screen Scraping with Macro Scheduler.

Screen Scraping via Optical Character Recognition

Macro Scheduler 14.4 includes some really neat functions which make it really easy to OCR a portion of the screen:

  • OCRScreen
  • OCRWindow
  • OCRArea

The first of these is the simplest. It simply scans the entire screen and returns all the text it can recognise. Of course this is also the slowest as it has to perform OCR against the entire screen. OCRWindow takes a window title and scans only the area of the screen where that window appears. This is nice and simple and a good compromise if the window isn’t too large. Finally, OCRArea can be given a rectangular screen region (X1,Y1,X2,Y2). You could use FindObject to find the coordinates of a specific UI object and pass those coordinates to OCRArea if you want to narrow things right down.

The Text Capture Functions

Macro Scheduler includes some Text Capture functions which can be used to extract text from a given window, rectangular screen area or screen point.  These functions use low level system hooks which monitor applications calling the various “TextOut” functions that Windows uses to output text to the screen.  By doing so they are able to capture this text.  The Text Capture functions return the text to  a variable which you can then use as needed.  

However, a few applications don’t use the Windows built-in functions to create and output text.  Don’t worry –Most do, but a few use their own techniques.  When you realise that text on the screen is just a sequence of small dots, if the application programmer decided to build his own routine to assemble text from dots rather than calling the Windows functions which already do that for you, you’re not going to be able to capture it.

The text capture functions and their limitations are explained here.  There is an example application, here, created with Macro Scheduler, which you can use to determine whether or not the text you want to capture can be captured using the text capture functions.

http://www.mjtnet.com/blog/2007/12/12/capturing-screen-text/
http://www.mjtnet.com/blog/2008/01/03/screen-scrape-text-capture-example/

Using the Clipboard for Screen Scraping

If the text you want to capture is selectable then you can use the clipboard to retrieve it.  A Macro Scheduler macro can send the keystrokes necessary to highlight and copy the text to the clipboard and then use the GetClipboard function to retrieve that text to a variable.  This is far less elegant than using the Text Capture functions but might be necessary if the application concerned is not utlising any of the Windows text out functions to create the text.

SetFocus>Notepad*
//Select ALL
Press CTRL
Send>a
Release CTRL
//Copy to clipboard
Press CTRL
Send>c
Release CTRL

//Get and display the data
WaitClipboard
GetClipboard>theData
MessageModal>theData
Older Posts »