When we are automating Windows applications we nearly always need to be able to detect new windows and wait for windows to open and close and be ready etc.  Unfortunately there are times when this is not possible.  Flash applications for example are just exposed as graphics and expose no windowed controls that we can grab a hold of.  Java applets are also a bit tricky as they use internal objects which Windows doesn’t know about.  Another tricky scenario is trying to automate Terminal Server or Citrix sessions at the client level.  The client just gets a graphical representation of the server’s screen.  There is no access to the objects that make up the user interface within the session unless you are on the server itself.  So we used to have to install Macro Scheduler on the server if we wanted to automate applications inside the session.  This is fine if we have access to install stuff on the server.  There are times when we don’t.

Another issue that came up recently was a customer that wanted to test the network lag time between a user entering some data into a Windows Terminal Server client and the server serving up the following page.  The conundrum is that if you install Macro scheduler on the server and make it wait for a new window there is no lag time since all the automation is happening locally to the server.  But on the client there is no way to know about the new window object.  Until now.

Enter the Macro Scheduler Image Recognition Library.   This includes functions that compare images and find images within larger images.  In other words we can compare screens and find objects within screens by searching for their image.   We can find and click on buttons and other controls and wait for the screen or parts of the screen to change based on what they look like.  This lets us automate applications graphically rather than having to know about windows and GUI controls.  In fact this is more analogous to how a user uses a computer.  The user watches the screen for changes in appearance and finds objects by looking at them with his eyes.  He looks at what he sees.  He doesn’t worry about window handles, x and y coordinates.  So the Image Recognition Library means we can automate any application regardless of the technology it uses to expose it’s interface.  It works at the graphical level of the screen.  You can download the Image Recognition Library at http://www.mjtnet.com/imagerecognition.htm

This entry was posted on Tuesday, January 3rd, 2006 at 4:43 pm and is filed under Scripting. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.