Marcus' Macro Scheduler Blog

As I have mentioned previously, Macro Scheduler 10 introduces some powerful new commands for capturing screen text. In this post I aim to explain what kinds of text can be captured with these new commands and why there will always be some text that cannot be retrieved.

First let’s look at how the existing functions, GetWindowText and GetObjectText work in Macro Scheduler 9.x and below.

Open up Macro Scheduler and click on the Tools menu and then the “View System Windows” option. You’ll end up with a window that looks something like this:

What we are looking at is a tree representation of windows open on the system. In the above screen shot the highlighted line is showing us an object of class “Button” with caption “Test Center”. Each line gives us the current handle of the object, followed by its class name and then its caption text, if any.

This caption text belonging to an object is made available to other processes – it is published if you will. An app can simply ask the control for its text by sending a simple message to it. That is what Macro Scheduler is doing when it builds this list of windows and objects. Macro Scheduler enumerates all top level windows and then for each one enumerates each of its child “windows”. Note that I use the term window interchangeably with object or control here – the controls that appear in the list are “windowed controls” – they have window handles. A handle allows us to interact with the control. If we know its handle we can send a message to it saying “please give me your text”. And so we get the text of the control back. This is what GetObjectText and GetWindowText do.

There are a number of shortfalls to this approach. One is that the “caption text” that the object publishes is not always the text that you see on the screen. In the case of standard Buttons, Edits, Windows and Checkboxes the published text is usually what you see. But other objects don’t necessarily work the same way. We usually know where we are with common controls – ones that belong to Windows, but custom controls in third party software may not follow the same rules. And a treeview’s caption, for example, is not the text belonging to all its nodes which is written to the screen. Furthermore not all text belongs to windowed controls. In Delphi applications, a control class called TLabel is commonly used as a way to write text on a window. These are often used to label other controls like edit boxes. But TLabels are not windowed controls – they don’t have handles. So this technique will not be able to retrieve their text.

We also can’t use this approach to get text from the likes of Word documents or Internet Explorer pages. This text is not just some simple caption property belonging to an ordinary control – it is created in a more direct way.

When Windows writes text to the screen it uses one of a number of functions deep within the Windows API. Most Windows applications will trigger these functions whether or not the programmer realises it. One such function is TextOut:

Windows GDI – TextOut
The TextOut function writes a character string at the specified location, using the currently selected font, background color, and text color.

Note that this function is part of gdi32 which is responsible for graphics – GDI = Graphics Device Interface. So TextOut is being called to “paint” a character to the screen.

With Macro Scheduler 10, when you call one of the new text capture commands Macro Scheduler uses a “hook” to listen in to calls to TextOut and other similar functions. It is therefore able to intercept what is written to the screen and retrieve the text output by a window.

This works with all kinds of applications including Microsoft Office, Internet Explorer, Firefox and the vast majority of everything else. There are still some exceptions though. Remember that this works by hooking these low level functions within Windows that are used to create text. The vast majority of Windows applications will use these system calls (often indirectly). However, some software may not. There’s no reason why a programmer can’t write text in an even lower level way – he might decide to paint a word pixel by pixel.

As an example – Java applications written with the AWT or SWT frameworks write text using Windows API functions. So we can detect text from those. But if you have a Java app produced with the Swing libraries, which handle text output their own way, you’re not going to be able to capture the text from it.

Finally, what about text on images? Well, text on an image was already there. It was painted by the artist. It is set in stone. So text that appears in a jpg, bmp or any other image file, cannot be detected with the new text capture commands, because it isn’t produced on the fly by one of Windows own text output functions.

The best way to determine whether or not the text you are seeing can be captured with the new text capture commands is to fire up the “Text Capture” sample macro. This will show you the text beneath your mouse cursor. So move the mouse over the text you are interested in and see if it gets displayed in the dialog. If it is, you know you can use the text capture commands to retrieve this text in your macro.

The only way to detect text that cannot be detected with the text capture commands is via OCR. Two methods to do this in Macro Scheduler are discussed here and here.

Something that comes up now and then is whether or not compiled Macro Scheduler macros can respond to hot-keys. Macro Scheduler itself lets you assign system-wide hot-keys to macros so that when you hit that key combination, no matter where you are in Windows, the macro will be fired.

Some people want to be able to distribute compiled macros (EXEs) to their friends and colleagues and have those EXEs respond to hot-keys in the same way. Something has to be running in order to listen for the hot-key and act on it. In the case of regular old Macro Scheduler that is Macro Scheduler itself. So how do we achieve it with a compiled macro?

Method 1 – make the EXE itself listen for the hot-key:

Use OnEvent to create a KEY_DOWN event handler to watch for the hot-key sequence:

OnEvent>KEY_DOWN,G,5,DoMacro

Label>Idle_Loop
Wait>0.02
Goto>Idle_Loop

SRT>DoMacro
..
.. Your Code Here
..
End>DoMacro

The above will run indefinitely and whenever CTRL+ALT+G is pressed the macro code in the DoMacro subroutine will be fired. Clearly the macro has to be running all the time to listen for the hotkey – after all, something has to listen for it. So you could have the EXE run on startup by adding a shortcut to the Startup folder for example. Perhaps also compile the EXE with the /NOSYSTRAY and /HIDE parms so that the EXE is not visible when it runs. Or, better, compile with a custom icon so that your EXE has it’s own fancy icon in the task bar.

See the documentation for the OnEvent function for more info on how to trap different keystrokes.

Method 2 – Let Windows do the work:

Don’t tell me – all these years using Microsoft Windows and you didn’t realise that keyboard shortcuts can be assigned to shortcuts?

So just create a desktop shortcut to your compiled EXE. Right click on the desktop shortcut and select properties. Notice the “Shortcut key” field. Press the keys you want to use to trigger the EXE. Press Ok. Sorted.

A short roundup of resources and ideas for using Macro Scheduler for system and software monitoring.

Event Log Monitoring

Here’s a tip showing how to create a script to monitor event log messages:
http://www.mjtnet.com/usergroup/viewtopic.php?t=4256

Process/Service Monitoring

This script demonstrates a function called IsProcessRunning to check if a specific process is running. Use it to create a monitor to watch a specific application process and make sure it is running. It could send an email if the process is no longer running. The same script shows some code to get the status of a service which could be used to create a service monitor.

Monitoring Files

Macro Scheduler has built in functions to see if a file exists, get file lists, get file dates, see if a file has changed, read from text files, read from INI files and more. See the IfFileExists, ReadFile, ReadLn, GetFileList, FileDate and ReadIniFile commands in the help file.

Monitoring Web/Intranet Resources

Use HTTPRequest to retrieve data from web resources, or FTPGetFile to download a file from an FTP server. Monitor email with ReceivePOP3.

Monitoring Windows and User Interfaces

If you just want to check that a window exists use IfWindowOpen. GetActiveWindow will return the active window title. Maybe you need an alert when a window contains a specific message, or piece of text. GetWindowText will retrieve all detectable text in a Window. Use GetControlText to retrieve text from a given control class and instance.

But what if you need to monitor a graphical element, or look for some non-detectable text? Use FindImagePos. The Image Recognition functions mean any kind of user interface can be monitored and/or controlled. You can also monitor applications running remotely (e.g. in Windows Terminal Server or Citrix sessions, or even non-Windows VNC apps) or in virtual environments.

One of our customers provides hosted software solutions to clients who require a high degree of up-time. Their clients’ businesses depend on their software remaining online. So they have used AppNavigator to create UI monitors that watch their applications and alert them if an interface goes down.

Alerts

The Event Log monitor example above uses email to alert the system administrator when an event is detected. Email is probably the most common method. Email can also often be used to send an SMS message (check with your mobile carrier). But an alert could just be a message box popup, or the script could run another application, write to a file or database or call a web resource. Pretty much any kind of alert could be created.

Have you used Macro Scheduler for system monitoring? If you have an example or story to share, please post a comment.