February 13, 2009

Image Recognition Common Mistakes

Filed under: Automation — Marcus Tettmar @ 12:16 pm

This is a copy of a reply I gave in the forums recently which I thought would be useful posted here.

It sometimes seems to surprise people that the only reason FindImagePos will fail to find an image on the screen is because …. that image is NOT on the screen! I’ve had people swear blind the image they are seeking is on the screen but when I’ve opened up the screen image and the needle image in Paint I can see big differences.

Here are some common mistakes I’ve seen that can lead to not finding the image you want Macro Scheduler to find (or rather, looking for the wrong image):

Mistake 1: Not using the built in Image Capture Tool. ALWAYS use the Macro Scheduler Image Capture tool. Other image editors may save with a different colour depth etc. To ensure like-with-like comparisons ALWAYS use the Macro Scheduler Image Capture Tool.

Mistake 2: Capturing the object when it is in a different state to how it appears at run time. E.g. buttons look different when they are focused, menus look different when they are selected, tooltips appear when the mouse is over objects, sometimes even window captions appear different when the window is not focused, sometimes objects look different when the mouse is hovered over them. Take all these things into consideration and make sure you capture the image as it will appear at the point in the process where you want to find it. You may need to use the delayed capture feature of the Image Capture Tool so that you can arrange focus before you capture the object. Consider making the script move the mouse to 0,0 in case the mouse ends up hovering over the button you’re looking for at runtime. I’ve seen buttons look different if the window is focused. If you have the window focused when you captured it you’ll need to make the script focus it before looking for it. Or vice-versa. Etc etc

Mistake 3: Capturing too much of the screen. FindImagePos compares a random sampling of pixels in the needle image and target area. So if you capture an area with lots of background it’s feasible that only the background pixels get chosen and you’ll end up with lots of matches. Be more specific in your captures, and if you need to, increase FIP_SCANPIXELS to make it scan more pixels. Update: in V12 you can set FIP_SCANPIXELS to ALL.

Mistake 3a: Thinking it didn’t find the image when in fact it DID find it, but it also found other possible matches. The function returns the number of images found and an array of matches. If you are clicking on the first match and it is not the right place, look at the number found. Consider what to do if number of matches is more than 1.

Mistake 4: Looking for the image at the wrong time – getting the order of events all wrong.

Some issues that can cause problems:

Graduated or patterned backgrounds. If you capture an object that is on a non-smooth background and have to capture some of the background as well, then if the object appears at a different position at runtime you may fail to find a match, because the background is not smooth. As with Mistake 3 above you probably just need to be more specific about what you capture. Or, if that isn’t possible, see if you can change the background. There’s no harm in making life easier for ourselves by setting the environment up to be a little more predictable so that we can automate/test more reliably. Don’t forget you can also change the colour tolerance parameter to handle small differences in colours.

Sometimes it’s helpful for diagnostic purposes to use ScreenCapture to capture the screen to a file prior to your FindImagePos call. Then, later, if it didn’t work as you expected you can open up the screen image and work out why.

See also: How to use Image Recognition.