Image detection failure possibly due to memory leak

rnoecker · Post by **rnoecker** » Fri Jul 19, 2013 5:35 pm

I've reported a few other related issues, but I think this one is different. I recently had another weird issue with image detection where the WaitScreenImage wasn't timing out as expected and then the FindImagePos was failing. I didn't think to check if NUM_FOUND was -1 which indicates a failure, but considering that a restart resolved the issue I think that is what was happening.

I noticed the memory usage had ballooned to 370 MB over the course of a few days in which I was running my script every ten minutes for testing (with gaps of hours here and there to account for failures and having to restart the script).

I'm thinking there must be a memory leak, and I've noticed before that the memory usage does increase over time, however, due to another stability issue that caused the program to crash (which I've reported) it never got to this point. Whereas it would always fail before 30 runs, I am not getting upwards of 40 runs, and when the script fails I am able to restart it which means it has run 150-200 times using the same process.

Environment
Windows 7
Resolution: approx. 1200x800 (running in a VM)
Color depth: 24bit
Script notes: does mostly image detection with some browser automation

Typical pattern
// open up report options
// reports top level menu
WaitScreenImage>%BMP_DIR%\menu_reports.bmp,%matchCoefficient%,CCOEFF
if>WSI_TIMEDOUT=FALSE
FindImagePos>%BMP_DIR%\menu_reports.bmp,SCREEN,%matchCoefficient%,1,XArr,YArr,NumFound,CCOEFF
If>NumFound>0
MouseMove>XArr_0,YArr_0
LClick
Else
GoSub>ErrorHandler,There was a problem accessing the reports menu
Endif
Else
GoSub>ErrorHandler,There was a problem accessing the reports menu
Endif

CyberCitizen · Post by **CyberCitizen** » Sat Jul 20, 2013 1:05 pm

What version, there was a memory leak discovered in the image detection, v14.0.12 was released to fix this issue, if you haven't got this version please update your MS.

rnoecker · Post by **rnoecker** » Sat Jul 20, 2013 2:50 pm

I am the one that reported that issue. I have 14.0.12e.

armsys · Post by **armsys** » Mon Jul 22, 2013 6:49 am

Hi Rnoecker,
Does your script include: Let>WSI_time=n
Try:

Code: Select all

Let>WSI_Time=30
WaitScreenImage>%BMP_DIR%\menu_reports.bmp,%matchCoefficient%,CCOEFF
If>WSI_TIMEDOUT=TRUE
 GoSub>ErrorHandler,There was a problem accessing the reports menu
 Exit
Endif
FindImagePos>%BMP_DIR%\menu_reports.bmp,SCREEN,%matchCoefficient%,1,XArr,YArr,NumFound,CCOEFF
If>NumFound>0
 MouseMove>XArr_0,YArr_0
 LClick
Else
 GoSub>ErrorHandler,There was a problem accessing the reports menu
Endif

rnoecker · Post by **rnoecker** » Mon Jul 22, 2013 12:53 pm

I assume you mean WSI_TIMEOUT. I do have the timeout being set. I'm not sure the connection, unless you think the problem is with my script. The script runs fine most of the time, and the problem was that it was finding a match when it shouldn't (not timing out too quickly) which I'm positive was because the image detection functions were failing internally.

LET>WSI_TIMEOUT=60

I just checked the current memory usage after a few days, and it is around 300 MB while no script is running. Can anyone give a legitimate reason why it would be using this much memory?

armsys · Post by **armsys** » Mon Jul 22, 2013 10:46 pm

rnoecker wrote:I assume you mean WSI_TIMEOUT...

Yes. Sorry for my typo.

rnoecker · Post by **rnoecker** » Wed Jul 24, 2013 3:59 pm

It looks like image detection is failing over time. It stops working and returns error codes every time. This is like the issue I reported with Windows 2003 except that would happen within the first few runs. This problem is happening over the course of a few days.

The memory usage is 130 MB which is a lot, but I have actually seen it as high as 300 MB.

The following block of code is the first that does image detection in my script. Once the image detection engine fails my script always fails at this point. It gets to the error message branch which is somewhere it should never be considering WaitScreenImage calls FindImagePos internally.

The call to WaitScreenImage returns:
WSI_RESULT = -1
WSI_TIMEDOUT = -1

The call to FindImagePos returns:
NUMFOUND = -1

WaitScreenImage>%BMP_DIR%\downloadOpenDialog.bmp,%matchCoefficient%,CCOEFF
If>WSI_TIMEDOUT=FALSE
FindImagePos>%BMP_DIR%\downloadOpenDialog.bmp,SCREEN,%matchCoefficient%,1,XArr,YArr,NumFound,CCOEFF
If>NumFound=1
MouseMove>XArr_0,YArr_0
LClick
Else
GoSub>ErrorHandler,There was a problem clicking the open button in the save dialog for the ica file used to launch the reports application
Endif
Endif

armsys · Post by **armsys** » Thu Jul 25, 2013 8:48 am

rnoecker wrote:The call to FindImagePos returns:
NUMFOUND = -1

Hi Rnoecker,
1. What's the size of downloadOpenDialog.bmp?
2. How did you create downloadOpenDialog.bmp?

rnoecker · Post by **rnoecker** » Thu Jul 25, 2013 12:50 pm

I really don't think it has anything to do with the image itself. This is just the first image detection code in the script so when the the image detection engine fails it fails. In fact, when it first fails it could be on different code (assumption), but after that it would fail on this code since it is the first in the script and that is when I am noticing it. I know this is a possible scenario since A) I've reported an issue like this before and B) The code works for 100+ times and then once this happens it fails every time.

I created all images using the image capture tool in the software.

Dimensions: 43x18
Size: 3.07 KB
Color Depth: 32 bit

armsys · Post by **armsys** » Thu Jul 25, 2013 1:38 pm

rnoecker wrote:I really don't think it has anything to do with the image itself. This is just the first image detection code in the script so when the the image detection engine fails it fails. In fact, when it first fails it could be on different code (assumption), but after that it would fail on this code since it is the first in the script and that is when I am noticing it. I know this is a possible scenario since A) I've reported an issue like this before and B) The code works for 100+ times and then once this happens it fails every time.
I created all images using the image capture tool in the software.
Dimensions: 43x18
Size: 3.07 KB
Color Depth: 32 bit

The figures you provided seem normal. That's, relatively small size.
I usually use SnagIt to capture images in bmp.
I usually keep the image size as small as possible to help the speed.
I use both WSI and FIP extensively.
Whenever possible, I avoid CCOEFF. EXACT search is always faster than CCOEFF.
I've never encountered the issue your described.

rnoecker · Post by **rnoecker** » Thu Jul 25, 2013 1:43 pm

I think the last bug was specific to CCOEFF matching. I'm using the fuzzy matching algorithm because the application I am working with is in a remote desktop window.

All of the images used in my script are small in size (ex. menu items, buttons, list items).

armsys · Post by **armsys** » Thu Jul 25, 2013 1:48 pm

rnoecker wrote:I think the last bug was specific to CCOEFF matching. I'm using the fuzzy matching algorithm because the application I am working with is in a remote desktop window.
All of the images used in my script are small in size (ex. menu items, buttons, list items).

Now I realize it's rather an issue related to the remote desktop window.
That would be tricky for both FIP and WSI to find the needle(s) on the remote dessktop window.

rnoecker · Post by **rnoecker** » Thu Jul 25, 2013 1:57 pm

Heh, no, as I said, the script works up until the point it doesn't. More specifically, it is an application run in a Citrix client so I don't want to rely on exact matching since I think it uses RDP internally.

Regardless of the difficulty in matching, those methods should never return -1 as that is an error code. If it was a matching issue the WSI image would timeout as expected.

I'm pretty sure the author knows about these things, I'm just trying to help (and make some noise) so that they hopefully get looked into. I like the idea behind the software, but I'm having trouble justifying committing money to licenses given the difficulties I've experienced. I've been struggling with the software for over a month now (image detection engine failure in Windows 2003, IEQuit instability, etc.).

Post by **Marcus Tettmar** » Thu Jul 25, 2013 3:59 pm

As you know we recently fixed a memory leak issue with FindImagePos and we did some pretty major stress testing then to prove it was fixed, and still run stress tests regularly but have so far not replicated any further issues. So I'm somewhat baffled at this stage and wondering what can be different.

Are you running this inside RDP? I wonder if RDP has something to do with it. Let me know what kind of environment you are running on and I will set up a stress test. I assume a script which loops and constantly Waits and looks for a known image would be an appropriate test?

rnoecker · Post by **rnoecker** » Thu Jul 25, 2013 4:34 pm

The script has maybe 20-30 similar image detection blocks. A lot of the blocks use the wait/find pattern as shown in my example. It takes about 5 minutes to run the script, and I have it running every 10 minutes for testing, although it may go for long periods where it isn't running due to an failure in the script (not related to this) until I start it back up again. I can try to pay more attention to the number of runs and number of days it is up before it happens, but I only recently became aware of this, and I've since experienced it twice.

Tangentially, assuming this pattern is required (I'm not aware of another way to do it), it would be nice if the WSI had the ability to return what FIP does (coordinates) to reduce duplication or FIP could have a timeout parameter (ex. keep looking for X seconds). It seems like a hybrid method is needed here to handle a streamed image where the quality isn't 100% or it may be slow to load compared to a local application (ex. you click a button but have to wait until the image updates fully to see a dialog that was created).

I use remote desktop to work on a server that is running the virtual machine (which shouldn't matter). The virtual machine is running Windows 7 and VirtualBox is used as the VM host. The VM is running in a window on the console session of the server so that I can see everything the script does (and to get around issues with using remote desktop to access the VM running the macro software). The VM is what is running the macro software, and nothing outside of the VM should affect the software since it is running on the console. The script launches an application in a Citrix client window from a website link (Citrix browser plugin). The application appears as a local application (i.e. it isn't in a remote desktop client), but what is being interacted with is an image which is streamed (and I think it uses the remote desktop protocol).

A lot going on, but I think it can be simplified as the only things I think matter are that it is a Windows 7 computer running the script which interacts with an application through the Citrix client.

As a test, my next step may be to create a test script that does nothing but run the example code block in a loop and see whether it fails. The rough part is that it seems to take days to happen.