Serious performance issue with FindImagePos

Technical support and scripting issues

Moderators: Dorian (MJT support), JRL

Post Reply
bossydog9
Junior Coder
Posts: 29
Joined: Wed Jan 08, 2020 12:46 pm

Serious performance issue with FindImagePos

Post by bossydog9 » Thu Jan 23, 2020 9:45 pm

Amongst the Macro Scheduler Blog pages is an article entitled "Ultra Fast Image Recognition and Text Capture Coming" and dated November 26, 2007. It is specific to Macro Scheduler 10 so version 14 should have the same, if not better, performance. Unfortunately, I haven't seen this kind of performance. The article says:
One of the most amazing improvements is the optimisation of the FindImagePos function. We’ve made a 30-50 fold speed increase. It is now almost instantaneous. Average time to locate a bitmap at the bottom right of a 1600×1200 24 bit screen on Vista running aero glass: 0.2 seconds.
On a fairly high spec Windows 10 SSD m/c, searching a 1024x768 window for a 37x29 image positioned at coordinates 180x403 (top left), the average time to locate the image was 1.08 seconds with the quickest being 1.01 seconds. The sample size was 264 searches.

One thing I noticed whilst testing the function was that after some time finding the images, it started to fail and returned -1 for the NumFound parameter. There was no code modification involved between these events. It started working again after I restarted the editor.

Unfortunately, this is a showstopper for the FindImagePos function so any help would be much appreciated.

User avatar
JRL
Automation Wizard
Posts: 3501
Joined: Mon Jan 10, 2005 6:22 pm
Location: Iowa

Re: Serious performance issue with FindImagePos

Post by JRL » Fri Jan 24, 2020 2:17 pm

Just did my own tests on a lower performing computer and a 1920 x 1080 screen using a 20 x 20 pixel needle. I didn't actually time the event because I used a program I already had written that doesn't have a timer, but the image is discovered pretty much as soon as I pick "Process". Far less than 1 second.
One thing I noticed whilst testing the function was that after some time finding the images, it started to fail and returned -1 for the NumFound parameter. There was no code modification involved between these events. It started working again after I restarted the editor.
What kind of results do you see if you don't run the search from the editor? The editor carries its own overhead and will pretty much always slow down the script and the Macro Scheduler functions.

User avatar
Marcus Tettmar
Site Admin
Posts: 7380
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: Serious performance issue with FindImagePos

Post by Marcus Tettmar » Fri Jan 24, 2020 6:32 pm

-1 means the image wasn't found. Most likely there was a subtle difference in appearance. Sometimes just one window going in or out of focus can have an effect. As JRL suggests, try running it outside of the debugger (editor). It will be faster and focus won't be affected by the editor/debugger.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

bossydog9
Junior Coder
Posts: 29
Joined: Wed Jan 08, 2020 12:46 pm

Re: Serious performance issue with FindImagePos

Post by bossydog9 » Tue Jan 28, 2020 9:31 pm

I've have done some further testing and have found that the speed is dependant on the source of the image, the haystack and the search type. The most dependable searches rely on the source of the image being the Image Capture Tool. The following shows the 100% success rate for 3 images captured by the tool along with the associated haystack, search type, elapsed time in seconds and the number of matches. Search type CCOEFF is markedly slower than search type EXACT. The EXACT Speed Factor shows how much faster it is:

Code: Select all

Summary Report - Images saved using Image Capture Tool								
								
Needle: image1.bmp							
CCOEFF Match Precision = 0.7				EXACT
EXACT Colour Precision = 0				Speed 
HayStack	Search Type	Elapsed	 NumFound	Factor
SCREEN		EXACT		0.1758		1	5.89
SCREEN		CCOEFF		1.0352		1		
WINDOW		EXACT		0.1563		1	6.57
WINDOW		CCOEFF		1.0273		1		
								
Needle: image2.bmp							
CCOEFF Match Precision = 0.7				EXACT
EXACT Colour Precision = 0				Speed 
HayStack	Search Type	Elapsed	 NumFound	Factor
SCREEN		EXACT		0.1719		1	6.50
SCREEN		CCOEFF		1.1172		1		
WINDOW		EXACT		0.1602		1	7.14
WINDOW		CCOEFF		1.1445		1		
								
Needle: image3.bmp							
CCOEFF Match Precision = 0.7				EXACT
EXACT Colour Precision = 0				Speed 
HayStack	Search Type	Elapsed	 NumFound	Factor
SCREEN		EXACT		0.1719		1	6.57
SCREEN		CCOEFF		1.1289		1		
WINDOW		EXACT		0.1484		1	7.34
WINDOW		CCOEFF		1.0898		1		
And whilst there isn't much in it, WINDOW/EXACT searches are always the fastest. CCOEFF searches are always the slowest. Images saved using the ScreenCapture function only have a 50% hit rate and, unfortunately, the hits are always the slowest CCOEFF searches:

Code: Select all

Summary Report - Images saved using ScreenCapture							
							
Needle: image1.bmp						
CCOEFF Match Precision = 0.7							
EXACT Colour Precision = 0							
HayStack	Search Type	Elapsed	 NumFound
SCREEN		EXACT		0.1680		0	
SCREEN		CCOEFF		1.0156		1	
WINDOW		EXACT		0.1602		0	
WINDOW		CCOEFF		1.0234		1	
							
Needle: image2.bmp						
CCOEFF Match Precision = 0.7							
EXACT Colour Precision = 0							
HayStack	Search Type	Elapsed	 NumFound
SCREEN		EXACT		0.1641		0	
SCREEN		CCOEFF		1.0000		1	
WINDOW		EXACT		0.1406		0	
WINDOW		CCOEFF		1.0078		1	
							
Needle: image3.bmp						
CCOEFF Match Precision = 0.7							
EXACT Colour Precision = 0							
HayStack	Search Type	Elapsed	 NumFound
SCREEN		EXACT		0.1797		0	
SCREEN		CCOEFF		1.0078		1	
WINDOW		EXACT		0.1641		0	
WINDOW		CCOEFF		1.0039		1	
The odd thing about these times is that the EXACT times are what you would expect for a successful search. Could it be that the NumFound value is wrong?

Also, what is the difference between a BMP created by the Image Capture Tool and the BMP created by the ScreenCapture function? Can't ScreenCapture produce the same BMP as the Image Capture Tool? Using the ScreenCapture function allows images to be captured programmatically, and accurately, it's just unfortunate that the far faster EXACT searches fail.

This is the code for the macro that I used for this testing:

Code: Select all

/*
This macro demonstrates the speed of the FindImagePos command using user-defined values for 
Match Precision and Colour Tolerance. 

Run it and browse for a bitmap file previously created with the Image Capture tool which is
currently showing on the screen. 

  Enter Match Precision value (0 to 1, default=0.7) for CCOEFF searches.
  Enter Colour Tolerance value (0 to 255, default=0) for EXACT searches.
  Hit Find to search for the image. 

A message box will be displayed showing elapsed time in seconds and the number of matches 
for the following search types:

  1. SCREEN - EXACT
  2. SCREEN - CCOEFF
  3. WINDOW - EXACT
  4. WINDOW - CCOEFF  
  
An Ask statement allows further images to be processed without re-starting the macro. 
Responding to the Ask statement with NO will display a summary report for all the images 
processed.

Change the Window Title at line 164 to suit
*/

VBSTART
VBEND

Dialog>Dialog1
object Dialog1: TForm
  Left = 441
  Top = 107
  HelpContext = 5000
  BorderIcons = [biSystemMenu]
  Caption = 'Dialog1'
  ClientHeight = 304
  ClientWidth = 368
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'MS Sans Serif'
  Font.Style = []
  OldCreateOrder = True
  ShowHint = True
  OnTaskBar = False
  PixelsPerInch = 96
  TextHeight = 13
  object msLabel1: TLabel
    Left = 18
    Top = 56
    Width = 45
    Height = 13
    Caption = 'Needle BMP File:'
    Transparent = True
  end
  object msLabel2: TLabel
    Left = 18
    Top = 16
    Width = 260
    Height = 13
    Caption = 'Browse for a BMP captured with the image capture tool'
    Transparent = True
  end
  object msLabel3: TLabel
    Left = 18
    Top = 112
    Width = 170
    Height = 13
    Caption = 'WINDOW: Match Precision: (0 to 1)'
    Transparent = True
  end
  object msLabel4: TLabel
    Left = 18
    Top = 256
    Width = 243
    Height = 13
    Caption = 'Create BMPs with Editor/Tools/Image Capture Tool'
    Transparent = True
  end
  object Label1: TLabel
    Left = 18
    Top = 184
    Width = 182
    Height = 13
    Caption = 'SCREEN: Colour Tolerance: (0 to 255)'
  end
  object Label2: TLabel
    Left = 18
    Top = 152
    Width = 168
    Height = 13
    Caption = '0 = low precision, 1 = high precision'
  end
  object Label3: TLabel
    Left = 18
    Top = 224
    Width = 156
    Height = 13
    Caption = 'Matched if colour is +/- tolerance'
  end
  object msEdit1: TEdit
    Left = 18
    Top = 72
    Width = 249
    Height = 21
    TabOrder = 0
  end
  object msButton1: tMSButton
    Left = 280
    Top = 72
    Width = 75
    Height = 25
    Caption = 'Browse'
    ParentShowHint = False
    ShowHint = False
    TabOrder = 1
    DoBrowse = False
    BrowseStyle = fbOpen
  end
  object msButton2: tMSButton
    Left = 280
    Top = 216
    Width = 75
    Height = 25
    Caption = 'Find'
    ModalResult = 10
    ParentShowHint = False
    ShowHint = False
    TabOrder = 2
    DoBrowse = False
    BrowseStyle = fbOpen
  end
  object msEdit2: TEdit
    Left = 18
    Top = 128
    Width = 129
    Height = 21
    TabOrder = 3
    Text = '0.7'
  end
  object msEdit3: TEdit
    Left = 18
    Top = 199
    Width = 129
    Height = 21
    TabOrder = 4
    Text = '0'
  end
end
EndDialog>Dialog1

AddDialogHandler>Dialog1,MSButton1,OnClick,DoBrowse

SRT>DoBrowse
  SetDialogProperty>Dialog1,MSButton1,DoBrowse,True
  GetDialogProperty>Dialog1,MSButton1,Filename,strFileName
  SetDialogProperty>Dialog1,msEdit1,Text,strFileName
END>DoBrowse

Let>strSummaryMessageText=Summary Report%CRLF%%CRLF%
Label>Start
Show>Dialog1,res1

If>res1=10
  //Change window title below to suit
  Let>strWindowTitle=SearchTimings.xlsx - Excel
  GetDialogProperty>Dialog1,msEdit2,Text,strMatchPrecision
  GetDialogProperty>Dialog1,msEdit3,Text,strColourTolerance
  ExtractFileName>strFileName,strShortFileName
  Let>strMessageText=Needle: %strShortFileName%%CRLF%
  Gosub>BuildMsg,CCOEFF Match Precision = %strMatchPrecision%
  Gosub>BuildMsg,EXACT Colour Precision = %strColourTolerance%
  Gosub>BuildMsg,
  
  Gosub>BuildSummaryMsg,Needle: %strShortFileName%
  Gosub>BuildSummaryMsg,CCOEFF Match Precision = %strMatchPrecision%
  Gosub>BuildSummaryMsg,EXACT Colour Precision = %strColourTolerance%
  Gosub>BuildSummaryMsg,HayStack%TAB%%TAB%Search Type%TAB%Elapsed%TAB%NumFound
  
  //SCREEN - EXACT
  VBEval>Timer,startTime
  FindImagePos>strFileName,SCREEN,strColourTolerance,0,PosX,PosY,NumFound,EXACT
  VBEval>Timer-%startTime%,Elapsed
  Format>%.4n,Elapsed,Elapsed
  Gosub>BuildMsg,SCREEN    EXACT%TAB%%TAB%Seconds Elapsed: %Elapsed% %TAB%NumFound: %NumFound%
  Gosub>BuildSummaryMsg,SCREEN%TAB%%TAB%EXACT%TAB%%TAB%%Elapsed%%TAB%%TAB%%NumFound%

  //SCREEN - CCOEFF
  VBEval>Timer,startTime
  FindImagePos>strFileName,SCREEN,strMatchPrecision,0,PosX,PosY,NumFound,CCOEFF
  VBEval>Timer-%startTime%,Elapsed
  Format>%.4n,Elapsed,Elapsed
  Gosub>BuildMsg,SCREEN    CCOEFF %TAB%Seconds Elapsed: %Elapsed% %TAB%NumFound: %NumFound%
  Gosub>BuildMsg,
  Gosub>BuildSummaryMsg,SCREEN%TAB%%TAB%CCOEFF%TAB%%TAB%%Elapsed%%TAB%%TAB%%NumFound%

  //WINDOW - EXACT
  VBEval>Timer,startTime
  FindImagePos>strFileName,WINDOW:%strWindowTitle%,strColourTolerance,0,PosX,PosY,NumFound,EXACT
  VBEval>Timer-%startTime%,Elapsed
  Format>%.4n,Elapsed,Elapsed
  Gosub>BuildMsg,WINDOW    EXACT%TAB%%TAB%Seconds Elapsed: %Elapsed% %TAB%NumFound: %NumFound%
  Gosub>BuildSummaryMsg,WINDOW%TAB%%TAB%EXACT%TAB%%TAB%%Elapsed%%TAB%%TAB%%NumFound%

  //WINDOW - CCOEFF
  VBEval>Timer,startTime
  FindImagePos>strFileName,WINDOW:%strWindowTitle%,strMatchPrecision,0,PosX,PosY,NumFound,CCOEFF
  VBEval>Timer-%startTime%,Elapsed
  Format>%.4n,Elapsed,Elapsed
  Gosub>BuildMsg,WINDOW    CCOEFF %TAB%Seconds Elapsed: %Elapsed% %TAB%NumFound: %NumFound%
  Gosub>BuildSummaryMsg,WINDOW%TAB%%TAB%CCOEFF%TAB%%TAB%%Elapsed%%TAB%%TAB%%NumFound%
  Gosub>BuildSummaryMsg,

  Let>MSG_STAYONTOP=1
  Let>MSG_CENTERED=1
  Let>MSG_HEIGHT=300
  Let>MSG_WIDTH=500
  MDL>strMessageText
Endif

Ask>Do you want to search for another image?,strAnswer

If>strAnswer=YES
  Goto>Start
Endif

Let>MSG_HEIGHT=600
Let>MSG_WIDTH=500
MDL>strSummaryMessageText

SRT>BuildMsg
  ConCat>strMessageText,%BuildMsg_Var_1%%CRLF%
END>BuildMsg

SRT>BuildSummaryMsg
  ConCat>strSummaryMessageText,%BuildSummaryMsg_Var_1%%CRLF%
END>BuildSummaryMsg

exit>0
One strange thing I noticed, a WINDOW/CCOEFF search returned a NumFound value of 1, even though the window title was incorrect. It found the image in the window it was actually in, not the window that I asked it to look in. The other 3 searches correctly returned 0.
Last edited by bossydog9 on Fri Feb 07, 2020 2:32 pm, edited 1 time in total.

User avatar
Marcus Tettmar
Site Admin
Posts: 7380
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: Serious performance issue with FindImagePos

Post by Marcus Tettmar » Wed Jan 29, 2020 2:30 pm

Your findings are exactly as expected and in accordance with all our documentation. Exact is a simple pixel by pixel comparison. CCOEFF is correlation coefficient which is a complex statistical pattern matching algorithm.

We only support the use of the built in capture tool. You can use other things if you want but since we then have no control over the pixel density and format you’re on your own. It should follow that we have created the capture tool to output images in the identical format we use for screen/window captures to ensure a proper comparison.

None of this should be new or surprising.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

Post Reply
Sign up to our newsletter for free automation tips, tricks & discounts