We have our first successful entry to the Scripts & Tips 10 Year Anniversary Competition. Robert White sent in this excellent Cookie Monitor script. It’s a nicely written script which is both very useful and also demonstrates effective use of a non-modal dialog and other scripting features such as looping, reading a list of files, string manipulation, and reading from the registry. The beauty of this script, as a demonstration script, is you can run it as-is and it will work on your machine without any modification. Grab the script here.

We’ve rewarded Robert with an extra year of upgrade protection. Well done Robert, and thanks for the script!

In my last post I showed you how to write an Image Recognition script in Macro Scheduler to automate a remote Citrix session. I demonstrated how to use Image Recognition functions to look for screen objects and create an Outlook email.

Well, in this demo I show you how to automate the exact same process using AppNavigator. The video is just over two minutes long and that includes the time taken to build the routine and play it back! You can see that there is no coding involved. Anyone who knows the process could create the routine.

Watch the video and compare to building the same process with code.

This is a very basic example where all we do is create an email. In the real world we’d take it further and we may also want to add failover checks to handle situations where window objects fail to appear or change. With the first solution we’d need to modify the code to handle this and our script would become much more complicated. With AppNavigator it would be a simple case of adding a not found action to the relevant images. We could tell it to try again, stop, jump to a different image, send an email (with a screen shot) to admin, run a script … etc. Maintenance becomes so much easier and less prone to introducing bugs compared to modifying long scripts of code.

What is Image Recognition?

Image Recognition allows Macro Scheduler to find a bitmap (needle) in another, larger, bitmap (haystack) and return its position. The larger bitmap could be a snapshot of the screen, and the smaller bitmap could be a capture of a toolbar button or some other screen object. Therefore Image Recognition allows us to locate objects on the screen, graphically. We can also wait for images to appear on the screen. So Image Recognition allows us to automate any kind of graphical interface. We can make the script wait for images to appear on the screen, find screen objects and therefore send mouse events to the correct part of the screen to focus or control the objects.

The functions most used for Image Recognition are as follows:

WaitScreenImage
FindImagePos
GetScreenRes
ScreenCapture

Building an Image Recognition Script

In this video I demonstrate the Image Recognition functions to automate Outlook running in a remote Citrix session. Traditionally automating interfaces in remote environments is difficult without having Macro Scheduler running on the remote server. Trying to automate a remote desktop with traditional commands is unreliable since the client doesn’t know anything about the windows and objects running on the remote server - all it sees is a graphical copy of the screen. This is why image recognition is so powerful in such situations.

Watch the video and then continue reading this article to learn how the script works.

In the video I first tell Macro Scheduler to find the position of a button on the screen. I first want the macro to find this button:

First I use the image capture tool to capture an image of this button from the screen and save it to a .bmp file. The image capture tool is available in the editor under the Tools menu, and also from the Capture button on the Command Builder dialog for the FindImagePos and other image recognition functions. I save the image to d:\citrix-images\new2.bmp.

The first thing I need my script to do is determine the dimensions of the screen. This is so that I can capture the entire screen later in the script. So I use the GetScreenRes function:

This returns the width and height of the screen in variables sX and sY. Next we capture the current screen to d:\screen.bmp using the ScreenCapture command:

So this captures the entire screen, from 0,0 to sX,sY returned by GetScreenRes.

Note: Since version 9.2 the above two steps are no longer necessary as it is now possible to specify SCREEN in FindImagePos and do not have to first copy the screen to a bitmap

Now we want the script to look for the new email button in this screen image using the FindImagePos function. This looks like this:

This looks for new2.bmp inside screen.bmp using a color tolerance of 20 (where 0 is not tolerant and the pixel colors must match exactly and 255 is highly tolerant and anything would match!). We set the fourth parameter to 1 to tell it to return the center coordinates of the image match. If this was zero it would return the top left position in screen.bmp where a match was found. But I would like to return the coordinates of the center position of a match. i.e. the center of the new email button on the screen. XPos and YPos are our variables the coordinates will be stored in. These are arrays. So the first match will be in XPos_0,YPos_0, the second match (if any) in XPos_1, YPos_1, etc. imgs is the return variable which is set to the number of matches found.

So now we can say “if a match was found, move the mouse to the first position and click”. This looks like:

Since we captured the entire screen, the coordinates returned by FindImagePos map directly to screen coordinates. If the ScreenCapture command had not captured the entire screen we would have to add the top x,y coordinates given in the ScreenCapture command as offsets to the MouseMove command. As we captured the entire screen x,y was 0,0 anyway. Sometimes it is not necessary to work with the entire screen and we could instead capture just a Window. Like this:

One would then have to add X,Y to any position returned by FindImagePos in order to map to an absolute screen position.

But back to the demo. We now have code which finds the new email button on the screen and clicks on it. This causes a new window to appear. If we were automating an application running on our desktop we could simply use the good old WaitWindowOpen command which is given the window title and waits for that window to appear. But, remember, this demo is automating a remote environment. The windows are on another computer. We don’t know about window titles. Citrix just sends us an image of the screen of the remote computer. So instead we will use the WaitScreenImage function. This is nice and easy:

I used the capture tool to capture an image of the top left part of the new window - showing the window icon and title:

That was saved to title2.bmp. So the above function simply watches the screen and waits until it finds this image on the screen. When that happens we know the new window has appeared.

By the way, you could create your own WaitScreenImage using a loop containing ScreenCapture and FindImagePos. This would give you more control and build in a timeout if required. But WaitScreenImage is a quick and easy way to wait for an image to appear on the screen without any further coding.

Finally, now the script knows the new window is present it can send some keystrokes, so we send some text to create the email. In the real world we’d probably use more image recognition to manipulate other objects on this new window and continue using the same techniques.

Tips

For performance reasons the image recognition functions do not check every single pixel in each image. That would take far too long. Instead a random sample of pixels is taken from the small “needle” image and each of these is checked against all the possible matching pixels in the larger “haystack” image. For this reason it is possible to get false positives if the image you have captured is not very specific. E.g. consider these two images:

These images are largely similar since we have too much gray area. If the random sample is all from the gray area both would match. These are from a status bar, and if we were searching for Done we could easily find Connecting. The solution here is to make the image smaller so that there is less of the gray background. Alternatively find something more specific on the screen.

AppNavigator Does it Without Code

AppNavigator takes all the coding away from image recognition and also adds some more power. It is clever enough to narrow down a search if it first finds too many matches, or none at all, and it also remembers the part of the screen a match was found in. Next time it will concentrate on that part of the screen and only search the entire screen if a match is not found. This increases performance and reliability. With AppNavigator you can capture screen objects and assign them actions without having to write code like that presented here. AppNavigator can be told what to do when an object can’t be found, and can also navigate the extraction of data from Excel or Access without coding that either.

To see how easy it is to create the same process with AppNavigator see my next post.

Update 12th October 2007: Macro Scheduler 9.2 now lets you specify SCREEN in the haystack parameter of FindImagePos. Therefore if you are scanning the entire screen you no longer need to use GetScreenRes and ScreenCapture to capture the screen to a bitmap file first. Just specify SCREEN in FindImagePos.

AppNavigator is a powerful new tool that lets you create automation routines graphically, without any need to write code. With this software you can create processes by highlighting screen objects and assigning actions to them. Any process can be automated based on simple graphics - by “seeing” the screen. Just show AppNavigator what to watch for, and what to click on, and build up a process based on visual cues. You can read more information here. We also have some Flash video demos available:

Creating an AppNavigator Process
Playing back an AppNavigator Process

To run WebRecorder, or VBScript macros which automate Internet Explorer in Vista (IE7), you will first need to disable “Protected Mode”. Consider this simple VBScript code:

Dim IE
Set IE = CreateObject(”InternetExplorer.Application”)
IE.Visible=1
IE.Navigate “http://www.mjtnet.com”

In IE6 and in IE7 under XP this code would create a new IE instance and then navigate it to mjtnet.com. But Vista adds something to IE called “Protected Mode” and for some reason Protected Mode causes the Navigate method to open up a new IE window. This is most annoying because it means any subsequent code in a script like the one above would fail because IE object refers to the instance created with the CreateObject call, but the page is now in the second instance.

You can disable “Protected Mode” in IE under Tools/Internet Options/Security. This puts things back to normal and means code like the above will continue to work.

One known issue with WebRecorder’s IEAuto.DLL is that the WaitNewIE function hangs when using IE7. We’re working on this now. This function is only used when an action in IE causes a new window/tab to appear. Apart from that, once you have disabled “Protected Mode” WebRecorder/IEAuto works fine with IE7/Vista.

Update: Version 1.78 is now available, fixing the issue with WaitNewIE on IE7.

Wells Anderson posts:

“Image Recognition is a brilliant new feature in 9.0. It allows you to write macros that press buttons that are “invisible” or have unpredictable locations. For example, add-ins to MS Outlook and other MS Office applications may have no menu equivalents and cannot be found using PushButton or FindWindowWithText. Image Recognition lets you find buttons and other screen elements and then perform actions on them, such as mouse clicks.”

He then goes on to share a great tip on how to deal with different font smoothing options. Check it out.

Macro Scheduler is able to automate other applications by simulating what a user does. There are some exceptions - e.g. it is possible to automate Internet Explorer and Microsoft Office applications through their COM interfaces. But for the vast majority of applications the only way to automate them is by simulating user input and manipulating their UI (User Interface) just as an ordinary user does. Therefore Macro Scheduler has to simulate a human to do what it does best.

Two people would have a hard time trying to use the same PC at the same time. Imagine Fred and Sally sitting in front of the same computer, both trying to use the keyboard at once. Fred is trying to write a letter in Word and Sally is writing an email in Outlook. At the same time. Can you imagine what would happen? Utter chaos is what would happen. Fred opens Word and starts typing. Sally grabs the mouse and opens Outlook. In an instant focus has switched away from Word and Fred’s keystrokes land elsewhere. Outlook hasn’t finished opening yet and for a brief while the Desktop was the active window. Fred’s keystrokes therefore landed on some icons and started opening other applications randomly. Nightmare. Miraculously they manage to sort that out and now Sally has Outlook open as the active window and starts typing. Fred quickly hits ALT-TAB to bring Word back and types some more of his letter. But Sally is still hitting keys on the keyboard so parts of her email end up in Fred’s letter. I think you get the picture. It’s more like a game of “Tug Of War”.

The only way two people could effectively use the same computer is to take turns. Either Fred writes his letter and then tells Sally she can write her email, or, if they want to do bits at a time it might work like this:

Fred: “I’m typing a letter in Word, please don’t touch the keyboard”
Sally: “OK, I’ll wait”
Time Passes
Fred: “OK, I’m done writing in Word for a while, your turn”
Sally: “OK, I’ve activated Outlook. Please don’t touch the keyboard until I say”
Fred: “OK, I’ll wait”
Time Passes
Sally: “OK, I’m done, your turn”

You get the idea. Now imagine that there’s a little red flag on top of the monitor. When Fred starts his letter he raises the flag. When he’s ready and happy to let Sally take over, he lowers it. Sally can now take over, and raises the flag. Essentially they are using a form of semaphore to coordinate their sharing of the computer.

Two Macro Scheduler scripts could do the same thing. Only instead of a little red flag on top of the monitor we could use a value in an INI file, or a registry entry. We’ll call this value RunStatus and it could be initialised to IDLE. Script A needs to focus a window and send some keystrokes to it. It first checks to see if RunStatus is IDLE. If not it waits until it is. It then sets RunStatus to ACTIVE and does what it needs to do before setting RunStatus back to IDLE. Script B meanwhile wants to activate Outlook and click on a toolbar button. It finds RunStatus is IDLE, so waits until Script A has set it back to ACTIVE, and so on. We might put these simple subroutines in both scripts:

SRT>WaitForIdle
  ReadIniFile>ctrl.ini,Share,RunStatus,IsIdle
  Wait>0.2
  If>IsIdle=ACTIVE,WaitForIdle
  EditIniFile>ctrl.ini,Share,RunStatus,ACTIVE
END>WaitForIdle

SRT>SetIdle
  EditIniFile>ctrl.ini,Share,RunStatus,IDLE
End>SetIdle

And then before any part of the script where we need to manipulate the User Interface we can call the WaitIdle subroutine:

GoSub>WaitForIdle

And when we’re done activating windows, or sending keystrokes, we can do:

GoSub>SetIdle

We’ve implemented a very simple form of semaphore for task communication in a multitasking environment. Using this approach we can allow two UI scripts to run concurrently without interfering with each other. If necessary, we can get a bit cleverer. If Script A must perform some automation only after the Script B has completed a specific task, we could create a different flag, or set the flag to a certain value. Script A waits for this value before it can continue.

Clearly it is expensive and wasteful to use one PC for each automation routine you need to run, so solutions like this save money and this is one approach you can use. But depending on the complexity of your scripts you may need to add lots of status checks and pauses. A much simpler solution is to use Virtualization. Install Microsoft Virtual PC, or VMWare and run each automation in its own virtual machine. Effectively you end up multiplexing the physical PC into several virtual PCs and your scripts can run safely in their own virtual machines without interfering with each other.

But on those occasions you need two UI scripts to run at the same time and interact with the same environment, our simple little flag hoisting solution is one you can use. Simple, but effective. But note that at this stage we haven’t considered file-locking and what happens if both scripts read or try to change the flag at exactly the same time. A more robust solution might be to use a value in a SQL database and take advantage of the database’s built in record locking.

A similar approach can even be used to control distributed scripts. With scripts on multiple networked computers, and the INI file in a network share (or using a SQL database), a controlling script can be created which modifies the flag. The networked scripts wait for the active flag before doing anything, and can be paused when the controlling script changes the flag. And flags can be used to synchronize scripts on different machines. This approach has been used successfully in load testing environments where hundreds of Macro Scheduler scripts are simulating tasks simultaneously to create load on the system being tested. You can read more about using Macro Scheduler for automated testing here.

As you probably already know, selecting a page of a tab control is usually best achieved by sending CTRL-TAB. So on opening an application we can use the following Macro Scheduler code to jump to the third tab:

SetFocus>Window_Title
Press CTRL
Press Tab * 2
Release CTRL

This issues CTRL-TAB twice and therefore cycles through two tab selections, leaving the third tab selected.

Usually when the window is first opened, the first tab is selected. But what if that is not always the case? How can we determine which tab is currently selected? This question came up in the forum recently.

To do this we can use the Windows API and send the TCM_GETCURSEL message to the tab control. When this message is sent to the control it returns the index of the selected tab.

Here’s an example. Right click on the desktop to open the Desktop Properties dialog and select one of the tabs to change the page. Then run this script:

//Initialise constants
Let>TCM_GETCURSEL=4875

//Get the handle of the Display Properties Window
GetWindowHandle>Display Properties,hwnd

//Find the handle of the SysTabControl32 (Tab Control) object
LibFunc>User32,FindWindowExA,tabctrl,hwnd,0,SysTabControl32,

//Send the TCM_GETCURSEL message to retrieve the selected tab index
LibFunc>user32,SendMessageA,tabindex,tabctrl,TCM_GETCURSEL,0,0

//Display the result
MessageModal>Selected Tab: %tabindex%

This finds the handle of the SysTabControl32 object on the Display Properties dialog and sends it the TCM_GETCURSEL message to retrieve the selected tab index.

You can use this approach with other applications. However, some applications use a different class of tab control, so you’d need to determine the class name of the tab control using the View System Windows tool in Macro Scheduler.

This one has bugged me for years - how to automate activating a system tray icon. For starters, the System Tray is designed to be manipulated with the mouse. We could send mouse clicks to icons in the system tray but for the fact that icons don’t always appear in the same place in the tray. Some systems contract and expand the system tray and on startup icons can be reshuffled and appear in a different order. So sending mouse events is no good.

Well it turns out there is a way to control the system tray with the keyboard. I hadn’t realised this. Thanks to Henk for discovering this. Henk demonstrates here that if you press the Windows key to open the start menu, then Escape to close the start menu keyboard focus is now on the Start button. If you then press Tab a few times focus ends up in the System Tray. If you hover over a System Tray icon you will see a Tool Tip appear above it showing the name of the application. Well the first letter of this text will select this icon. So once you’ve tabbed into the System Tray you can now press the letter key corresponding to the first letter of the icon’s title and it will be selected. Then pressing Enter will activate it.

The number of tabs you need to send to get to the System Tray depends on how you have your task bar set up. If you have the QuickLaunch toolbar enabled you’d need an extra Tab. And in XP with the contracting/expanding System Tray you need to press the right arrow key to move off the contract/expand button. So this approach is not entirely portable but will work on your system.

So I gave this some more thought and came up with a portable approach which focuses the System Tray area directly - without having to open the Start menu and tab around. My approach uses the Win32 API functions FindWindow and FindWindowEx to find the handle of the System Tray Notification Area and then focus it directly. Here’s the code:

Let>WIN_USEHANDLE=1
LibFunc>User32,FindWindowA,h1,Shell_TrayWnd,
LibFunc>User32,FindWindowExA,h2,h1,0,TrayNotifyWnd,
LibFunc>User32,FindWindowExA,h3,h2,0,SysPager,
LibFunc>User32,FindWindowExA,h4,h3,0,ToolbarWindow32,Notification Area
SetFocus>h4
//Change to first letter of icon to activate
Send>v
//If more than one with same first letter, then repeat above line for each
Press Enter

Thanks again to Henk for the tip. See the forum post here.

Here’s a script to automate the installation of Adobe Reader:

http://www.mjtnet.com/usergroup/viewtopic.php?t=3034

This script demonstrates how to wait for windows to change based on the text within them. As with the majority of installers, Adobe Reader’s installer has a series of windows with the same title with differing content, and “Next” buttons to progress through the installation. Since the window title remains the same we can’t wait for the next window based on the window title so we must wait for the content to change. My install script demonstrates use of the FindWindowWithText function inside loops to show how to wait for specific text within the window. Using this approach we can wait for each step in the install to complete.

An alternative method that some might prefer is to use a graphical approach using the Image Recognition Library. Using this we could write a script that waits for an enabled “Next” button and clicks on it when it is found, for example.