We have today released WebRecorder 3.0.

This is a major rewrite with a better recording engine capable of recording more tags and producing leaner, cleaner, code and a faster and more reliable runtime.

It also introduces a download manager so that file downloads can be recorded and scripted more easily without recourse to sending keystrokes to IE’s standard file download box.

We’ve improved the existing commands so that you can script more tags and use more attributes and added new functions for retrieving data, retrieving the screen position of elements and interfacing with the new native IE functions in Macro Scheduler 13.1.

Oh, and you no longer need to use the mouse to record clicks and form fills!

• Cleaner, leaner, faster code.
• More reliable method for waiting for documents to complete loading
• Download manager for scripting file downloads
• Better error trapping
• Ability to set timeout for Clicks and Form Fills etc
• Click and Form fill functions will wait until target tags exist (within timeout)
• New functions for extracting data
• New function to get X,Y screen position of elements
• No need to use the mouse to identify recorded elements
• New, improved UI look and feel
• Runtime component integrates better with Macro Scheduler’s script controller (faster termination, less liable to hang ups when aborting scripts etc)
• Interface easily with Macro Scheduler’s IEGetTags and IETagEvent function
• New script function for waiting for text in the page before continuing
• Functions to automatically set extracted tag buffer size and avoid buffer overruns

More information on WebRecorder here.  WebRecorder 3.0 scripts require Macro Scheduler 13.1 to run.

Enterprise customers with in-date maintenance can already download WebRecorder 3.0 from the registered download area.

Other customers can view upgrade options in their registered download account, or download a trial version here.

Trial Downloads | Registered Downloads | Upgrades

Related posts:

1. Special Offer – Get WebRecorder Free
2. Macro Scheduler 13.1 Update
3. Calling Macro Scheduler Functions from PowerShell

I’ve seen quite a lot of requests lately from people wanting to know how to extract text from web pages.

Macro Scheduler’s optional WebRecorder add-on simplifies the automation of web pages and includes functions for extracting tables, text or HTML from web page elements. WebRecorder’s Tag Extraction wizard makes it easy to create the code.

Sometimes you can choose a specific HTML element and identify it uniquely via it’s ID or NAME attribute. But other times you might want all the text from the whole page, or you may need to extract the entire page and then parse out the bits you’re interested in using RegEx or some other string manipulation functions.

To extract an entire page I specify the BODY element. If you want to extract data from web pages it does help if you know a little about HTML. And if you do you’ll know that each page has just one BODY element which contains the code making up the visible portion of the page.

Here’s code produced using WebRecorder when navigating to mjtnet.com and using the Tag Extraction wizard to extract the BODY text:

IE_Create>0,IE[0]

IE_Navigate>%IE[0]%,http://www.mjtnet.com/,r
IE_Wait>%IE[0]%,r
Wait>delay

//Modify buffer size if required (you may get a crash if buffer size too small for data) ...
Let>BODY0_SIZE=9999
IE_ExtractTag>%IE[0]%,,BODY,0,0,BODY0,r
MidStr>r_6,1,r,BODY0

MessageModal>BODY0

The macro simply displays just the text in a message box but could be set to pull out the full HTML. You could then parse it with RegEx to get the information you are interested in.

You will need WebRecorder installed for the above to work.

If you don’t have WebRecorder you can do the same with a bit more work using VBScript. Some library functions for doing this can be found here and here.

So here’s the equivalent in VBScript:

VBSTART
Dim IE

'Creates IE instance
Sub CreateIE
  Set IE = CreateObject("InternetExplorer.Application")
  IE.Visible=1
End Sub

'Navigate to an IE instance
Sub Navigate(URL)
  IE.Navigate URL
  do while IE.Busy
  loop
End Sub

'This function extracts text from a specific tag by name and index
'e.g. TABLE,0 (1st Table element) or P,1 (2nd Paragraph element)
'set all to 1 to extract all HTML, 0 for only inside text without HTML
Function ExtractTag(TagName,Num,all)
  dim t
  set t = IE.document.getElementsbyTagname(Tagname)
  if all=1 then
    ExtractTag = t.Item(Num).outerHTML
  else
    ExtractTag = t.Item(Num).innerText
  end if
End Function
VBEND

VBRun>CreateIE
VBRun>Navigate,www.mjtnet.com

VBEval>ExtractTag("BODY",0,0),BodyText
MessageModal>BodyText

But what if you already have a macro which already opens IE, or works against an already open instance of IE? The above macros need to create the IE instance before they can access them and extract data from them. You may have a macro that already starts IE some other way – maybe just by using a RunProgram or ExecuteFile call, or indirectly via some other application. Many times people tackle the extraction of data from such an IE window by sending keystrokes to do a Select-All, Edit/Copy and then use GetClipboard; or even File/Save As to save the HTML to a file. This of course adds time and can be unreliable. So how else can we do it?

Well, this tip shows us a function we can use to attach to an existing IE instance. So let’s use that and then use our ExtractTag function to pull out the BODY HTML:

VBSTART
Dim IE

' Attaches to an already running IE instance with given URL
Sub GetIE(URL)
  Dim objInstances, objIE
  Set objInstances = CreateObject("Shell.Application").windows
  If objInstances.Count > 0 Then '/// make sure we have instances open.
    For Each objIE In objInstances
      If InStr(objIE.LocationURL,URL) > 0 then
        Set IE = objIE
      End if
    Next
  End if
End Sub

'This function extracts text from a specific tag by name and index
'e.g. TABLE,0 (1st Table element) or P,1 (2nd Paragraph element)
'set all to 1 to extract all HTML, 0 for only inside text without HTML
Function ExtractTag(TagName,Num,all)
  dim t
  set t = IE.document.getElementsbyTagname(Tagname)
  if all=1 then
    ExtractTag = t.Item(Num).outerHTML
  else
    ExtractTag = t.Item(Num).innerText
  end if
End Function
VBEND

VBRun>GetIE,www.mjtnet.com

VBEval>ExtractTag("BODY",0,1),BodyHTML
MessageModal>BodyHTML

This snippet assumes a copy of IE is already open and pointing to www.mjtnet.com. The GetIE call creates a link to that IE window and then we use the ExtractTag function to pull out the HTML of the BODY element.

These examples use the BODY element, which will contain everything displayed on the page. As I mentioned before you can be more specific and specify some other element, and with WebRecorder, or a modified version of the ExtractTag VBScript function use other attributes to identify the element (the existing VBScript ExtractTag function shown above just uses the numeric index). WebRecorder tries to make it simple by giving you a point and click wizard, making some assumptions for you, so that you need not fully understand the HTML of the page. But it still helps you understand HTML. Looking at the source of the page you should be able to identify the element you need to extract from. And whether you extract directly from that or extract the BODY and then use RegEx being prepared to delve into the HTML source is going to get you further.

UPDATE: 19th January 2012

As of version 13.0.06 Macro Scheduler now includes a function called IEGetTags. For a given tag type and IE tab this will retrieve an array of tag contents. It can extract just the text, or html of the tags. This example extracts the inner HTML of all DIV elements in the open IE document currently at www.mjtnet.com:

IEGetTags>mjtnet.com,DIV,H,divArr

You can then cycle through each one with a Repeat Until

If>divArr_count>0
  Let>k=0
  Repeat>k
    Let>k=k+1
    Let>this_div_html=divArr_%k%
    ..
    .. do something with it
    .. e.g. use RegEx or substring searching to determine
    .. if this is the DIV you want and extract from it
    ..
  Until>k=divArr_count
Endif

To further identify the tag you are interested in, or find the data you want, you can use RegEx, EasyPatterns, or string functions.

Macro Scheduler 13.0.06 and above also has a function called IETagEvent which will let you simulate a Click on a given tag, focus it, or modify its value. So once you have identified a tag using IEGetTags and your Repeat/Until loop you can click on it, focus it or modify its value (e.g. for form fields).

Related posts:

1. WebRecorder 3.0 – Faster, More Reliable, More Functions

As noted yesterday I have been waiting on Twitter to provide xAuth access. They declined, saying it was not appropriate. I’m not really sure why.

No matter, I decided to make a small DLL to simplify Tweeting from Macro Scheduler. It uses the full oAuth interface.

Implementing oAuth in Macro Scheduler code would require lots of VBScript code and would be very complicated (although doable in theory). So instead I decided to create a DLL which you can use in Macro Scheduler to tweet in one line of code.

You can download it here.

And then to post a status update all you need to do is something like this:

Let>message=Hello from Macro Scheduler
LibFunc>%SCRIPT_DIR%\tweetlib.dll,UpdateStatus,r,message,buff,1024

Note that the first time you call UpdateStatus you will be asked to log into Twitter and click “Allow” to authorise Macro Scheduler to access your account. You will then be given a PIN to enter. You only need to do this once. If you ever need to revoke access and start over call the RemoveCredentials function. Your Twitter username and password are NOT stored anywhere. This uses the oAuth authorisation scheme which provides an access token. It is the access token which is stored and this only allows Macro Scheduler to access the API for your account.

The return buffer will contain the XML of the status update operation if successful or an error message if not.

See readme.txt and sample .scp in the zip file. Enjoy.

Related posts:

1. Scraping Data From Web Pages

A while back I posted an article showing how to Tweet via Twitter‘s API. It uses basic authentication which Twitter plan to turn off in the near future. The alternative, oAuth is awkward for desktop based apps, but xAuth is now available and should be doable in Macro Scheduler. I have requested xAuth access from Twitter and, assuming it’s doable, will try and provide an example once I’ve received it and tried it out.

In the mean time it occurred to me that we don’t really need an API if all we want to do is send a status update. We can do that easily using Macro Scheduler and WebRecorder functions by controlling an instance of Internet Explorer, which can be done in the background.

Below is a script which demonstrates this. It offers a function called LoginToTwitter which need only be called once per session, and an UpdateStatus function to update your status. Just set your Twitter username and password in the first two lines and you should be all set.

Let>TW_USERNAME=XXXXX
Let>TW_PASSWORD=XXXXX

//only need do this once per session
GoSub>LoginToTwitter

GoSub>UpdateStatus,This is a test

GoSub>UpdateStatus,This is a test 2

GoSub>LogOut

// END

//*** SUBROUTINES ***//
SRT>LoadWR
  //load the WebRecorder runtime
  LibLoad>IEAuto.dll,hIE
  If>hIE=0
    MessageModal>Could not load IEAuto.dll, make sure it is in the path or edit the LibLoad line.
    Exit>0
  EndIf
END>LoadWR

SRT>LoginToTwitter
  GoSub>LoadWR
  //open IE
  LibFunc>hIE,CreateIE,ieTwitter,0
  LibFunc>hIE,ShowIE,res,ieTwitter,1

  //log in to Twitter
  LibFunc>hIE,Navigate,r,ieTwitter,http://twitter.com/login
  LibFunc>hIE,WaitIE,r,ieTwitter
  LibFunc>hIE,FormFill,r,ieTwitter,,,session[username_or_email],TW_USERNAME,0
  LibFunc>hIE,FormFill,r,ieTwitter,,,session[password],TW_PASSWORD,submit

  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
END>LoginToTwitter

SRT>UpdateStatus
  LibFunc>hIE,FormFill,r,ieTwitter,,,status,UpdateStatus_VAR_1,submit
  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
END>UpdateStatus

SRT>LogOut
  LibFunc>hIE,KillIE,r,ieTwitter
END>LogOut

You need the IEAuto.DLL library which is installed with WebRecorder.

For a bit of fun the following code copies the currently highlighted text to the clipboard and tweets it. So assigned to a hot key it can be used to tweet any text from any application.

Press CTRL
Send>c
Release CTRL
WaitClipBoard
GetClipBoard>theText
GoSub>UpdateStatus,theText

The real challenge is finding something useful to do with it!

Related posts:

1. Macro Scheduler 13.1 Update
2. Calling Macro Scheduler Functions from PowerShell
3. Macro Scheduler 13.0.03 Update

One of my Windows XP systems installed an automatic update last night when I shut it down.  This morning I was presented with this:

As I understand it this is as a result of an EU competition law decision.   Is this a good thing or a bad thing?

On the one hand it seems a waste of money and illogical.

I’m no fan of IE, but why shouldn’t Microsoft build a web browser and make it part of their operating system?  Should the EU force them to offer a choice of text editors, calculators, paint programs and calendar applications too? Where should it end?

Plus, yet another dialog like this is surely going to confuse the ordinary user.  Many have no idea what a “web browser” is and just call it “My Internet” or “Google”.  The dialog preceding the one above has to go to the trouble of explaining what a web browser is.  And isn’t the user just going to take the first option anyway?  In which case should the EU force Microsoft to make IE8 something other than the default option?

But then if I were trying to sell a web browser I guess I would welcome this decision. It might give me a little extra exposure.

What do you think?

Related posts:

1. Authenticate Your EXEs – Discounted Code Signing
2. Why Can’t I Colour My Dialog Buttons?

Here’s a quick and simple proof of concept for running Macro Scheduler macros via the web and having their output displayed in the user’s browser:

The screenshot shows the PHP script, Macro Scheduler script and Internet Explorer being used to run the macro.

1) If you don’t already have a Windows based web server with PHP running, download and install WAMPServer. It’s easy.

2) Create a simple PHP script which takes an EXE name as a parameter and any parameters you want to pass to it. The following script will run EXEs that are in the c:\wamp\ folder, passing in any parameters provided and will dispay the EXEs output.

<?php
$exe = $_GET['exe'];

$dir = "c:\\wamp\\";

$parms = "";
foreach($_GET as $key=>$val) {
  $parms .= "/$key=$val ";
}

echo shell_exec("\"$dir$exe\" $parms");
?>

3) PHP’s safemode must be disabled for this script to work.

4) Create a Macro Scheduler macro and use SOWrite or SOWriteLn to output information. Compile it with the “Create Console App” option checked. Compile the EXE (or copy it) to the c:\wamp\ folder.

5) Now the macro can be executed via the web using http://servername/runmacro.php?exe=my.exe&parm1=value&etc=… which could be a link or entered into the browser directly.

6) Consider adding further security to the script to prevent anyone running any EXE on your server, or putting it in a password protected folder. I’ll leave that to you.

Macro Scheduler Enterprise comes with the msNet Remote Controller which includes a CGI module for running Macro Scheduler macros via web servers.

Related posts:

1. About the MacroScript SDK – How to Run Macro Scheduler Code Within Your Own Applications
2. Calling Macro Scheduler Functions from PowerShell
3. Macro Scheduler 13.1 Update

Way back in the deep and distant past when the Internet was new and Bill Gates thought it was just a passing fad, I remember reading about a Cola vending machine on a University campus that some frivolous young boffins hooked up to the ‘net so that you could check its inventory from anywhere in the world using an old fashioned network command called “finger”. Why? Because they could.

Fast forward to the technologies of the current day and the latest trend of Twitter, and history is repeating itself. In the last week I’ve read about a restaurant that can take orders via Twitter, a bakery tweeting the emergence of fresh loaves from the oven; and, utterly pointless, some guys who created a system which sends a tweet every time their cat enters or exits its cat flap. Why? Well, because they can I guess.

Not wanting to be left out I decided to write some Macro Scheduler code to tweet status updates and monitor replies. Why? Well there might be a good reason for being able to do this – I’m sure someone will have one. Perhaps you have a client who wants you to set up a system to monitor the movement of his cat, process restaurant orders, or your local baker wants an automated fresh-loaf tweeter! But mostly, it’s because we can.

You’ll find the Twitter API documentation here. Here’s the code to Tweet a status update:

Let>username=YOURTWITTERNAME
Let>password=YOURPASSWORD

//Tweet from Macro Scheduler
Let>url=http://%username%:%password%@twitter.com/statuses/update.xml
Let>message=Kitty just left the buildng
HTTPRequest>url,,POST,status=%message%,result

Being serious for a moment I can see how a macro that monitors an application might want to post status updates to Twitter, or a backup script could alert you by Twitter when there’s a problem. It might be a public system, but don’t forget that Twitter profiles can be made private too, and Tweets can be viewed on and sent from your BlackBerry, iPhone, or even by SMS.

The following script sets up a loop which monitors your Twitter stream for “mentions” of your username. This might form the basis of a script which retrieves orders. Perhaps it could listen to Twitter for commands and carry out actions based on what message was sent. Or perhaps you just want a macro which does something when a cat decides to head out for the night. Use your imagination.

Let>username=YOURTWITTERNAME
Let>password=YOURPASSWORD
Let>ini_file=%SCRIPT_DIR%\twit.ini
Let>_delay=30

VBSTART
VBEND

//monitor twitter username "mentions" loop
Label>monitor_loop

Let>url=http://%username%:%password%@twitter.com/statuses/mentions.xml
HTTPRequest>url,,GET,,result

//remove the  portion (I don't need it and it avoids distinguishing the text IDs from the user IDs.
RegEx>[^>](.*?),result,0,user_matches,nf,1,,result

//extract all texts
RegEx>(?<=)[^>]*?(?=),result,0,text_matches,num_texts,0
If>num_texts>0
  //extract all ids
  RegEx>(?<=)[^>]*?(?=),result,0,id_matches,num_ids,0

  //get last known
  Let>last_known_id=0
  IfFileExists>ini_file
    ReadIniFile>ini_file,SETTINGS,LAST_ID,last_known_id
  Else
    WriteLn>ini_file,wlnr,
  Endif

  //iterate through texts
  Let>k=0
  Repeat>k
    Let>k=k+1
    Let>this_id=id_matches_%k%
    If>this_id>last_known_id
      Let>msg_text=text_matches_%k%
      /*
      msg_text contains the message
      Use your imagination here!
      For now we'll show it in a message
      */
      MessageModal>msg_text
    Endif
  Until>k=num_texts

  //store last ID
  EditIniFile>ini_file,SETTINGS,LAST_ID,id_matches_1
Endif

Wait>_delay
Goto>monitor_loop

The script retrieves the 20 most recent “mentions”. It stores the last seen ID in an INI file so that on the next check it ignores those it has seen before, only retrieving messages with a larger ID number.

This is a quick and dirty solution with no error checking, using RegEx to parse the XML that is returned by the call to Twitter. You may prefer to use the MS XML object as shown here.

Whether this proves useful or completely pointless, I hope you have fun. If you’re using Macro Scheduler with Twitter, please add a comment below to let us know how … and why!

Don’t forget you can follow me on Twitter where I may occassionally say something useful.

Related posts:

1. Macro Scheduler 12.1.6 – Now With Macro Recording in Script Editor
2. Calling Macro Scheduler Functions from PowerShell
3. Macro Scheduler 12.1.7 Available

Some of you may remember that we had some issues with Internet Explorer 6 support when we released our new web site. I made some negative comments about IE6. And I was being polite. Unfortunately IE6 is still in use in millions of companies and, as I mentioned in that last post, some 35% of our web traffic is delivered by people using IE6. If it were up to me I’d banish it to web browser hell.

So I’m all for a new campaign from .Net magazine: Bring Down IE6.

If you have the power to do so, upgrade now, or try out Firefox or Google Chrome . If you’re working for a company that locks down their PCs and forces you to use IE6, get vocal – educate your colleagues and IT department. Petition them to lift you out of the dark ages!

Check out the site to see how you can get involved.

No related posts.

If you’re one of those who has heard of Twitter but just doesn’t quite “get it” yet you should find Scott Hanselman’s How To Twitter – First Steps and a Twitter Glossary useful. It explains what Twitter is, why it’s useful and how to tweet.

I didn’t get it at first. I wasn’t quite sure what I would get out of it. Most people who now use Twitter regularly were probably the same to begin with. Now I realise how useful it is. I follow friends, people and companies that I find interesting and I want to stay up to date with. I follow news sites like the BBC and CNN. I probably get most of my news via Twitter now (In our house the TV seems to be permanently tuned to Nick Jr these days!). Most of the bloggers I read now tweet their blog updates, so I no longer need to monitor their RSS feeds.

It’s useful. It’s fun too. Even if you don’t have anything to say, you can get a lot out of following people that interest you. And, as Scott says, the nice thing is if you want to respond you can just jump in and reply to anyone.

So if you’ve been wondering what all the fuss is about, read Scott’s blog post, jump in and join the conversation. You can start by following me. My twitter ID is http://twitter.com/marcustettmar

Related posts:

1. What happened to good manners?

I see IE7 has been released. Haven’t installed it yet, but I do have an early beta version running on another PC and found that a bit clunky. I’m a Firefox fan though, so I’ve just downloaded and installed Firefox 2 RC3. I like it. The interface has been given a subtle update but it is also noticeably faster. I really like the built in spell check. I used to use the Google toolbar to spell check my posts, when I remembered – I invariably forgot! So having spell check built in is a great improvement. Now I just need to change the dictionary from US English to British English!

You can download Firefox 2 RC3 here.

Related posts:

1. Comparing Old and New – Why Upgrade?