Optimising bitmap detection

[geoffwoz]

We would like to optimise the bitmap detection capabilities of one of our scripts, as the timing of the arrival of the bitmap is important to us, while not wanting to overload the CPU on the system being monitored.

The issue we see is, when running an image detection action to wait for bitmap to appear, there is no way for us to limit the area to be scanned, it is all of the screen or nothing. (WaitScreenImage)

If we use the wizard to "find position of an image to interact with", then we are presented with a compound script that captures a specific portion of the screen/window to a local file, and then executing the find bitmap on the file. This mechanism doesn't explicitly include a "looping wait" mechanism like the full screen wait for bitmap option. I could potentially just put this in a loop, but it seems wasteful to capture constantly to a bitmap file to then read in, when all that I really need is the window scanning function to simply limit the scanning to a subset of the screen size.

Would it be possible to enhance "WaitScreenImage" to include a window handle, or ideally a range of screen cooordinates that would be scannned ? This would minimise the amount of scanning to be performed. reducing CPU, and also allowing for more timely detection.

[hagchr]

Hi,

For FindImagePos> you can select bitmap for both needle and haystack (so could be smaller areas). So I guess it should be possible to set up a custom event that triggers when the image appears (ie a custom WaitScreenImage). If the image always appears at the same position you also have CompareBitmaps> and GetRectCheckSum> that could possibly be used creatively for small areas to trigger events etc.

[geoffwoz]

Thanks for the quick reply.

Perhaps I'm missing something, but if I use findImagePos then when scanning the "SCREEN" haystack my understanding is that it will scan the whole screen (or potentially window, but in my case the window is full screen, so much the same), the only way that I can get a subsection of the sreen scanned with FindImagePos is I have to combine it with ScreenCapture to a local TMP file. I would also have to call it multiple times with a delay (min of 1 second wait). So it won't give me very good granularity on time, and requires the subset of the screen to be copied to a file for processing.. just seems a bit of a bandaid.

I've looked a bit closer, and I guess that I could use a loop containing WaitRectChanged and then call screencapture and then call FindImagePos to confirm if the image matches... alternatively If I could work out a way to easily generate a checksum from a PNG I could then use that in GetRectCheckSum.

I suppose either would work, but it just complicates things, when I wonder if providing a bounding rectangle to the FindImagePos would be a cleaner solution, that could be performed faster, and without the need for "external" looping in the macroscript.

[hagchr]

Hi,

I agree, the best would be if one could choose a search area for the WaitScreenImage>. Not sure if it helps but I have one example of a custom SRT> that waits for an image in a defined rectangle, and clicks on it when found.

In the example the SRT is called 3 times, first you look for needle1 in area 1, then needle2 in area 2, and finally needle3 in area 3 (for each one you also click on the needle).

In my example the image I search for gets dimmed into the screen so I need to add 0.5s between the SRT calls to make it robust.

As you mention one still have to go the route via screen capture but you can slow the onEvent iterations down by setting WIN_SLEEP=1, and you can avoid sending the Screen Capture to the clipboard by setting SCREENCAP_CLIPBOARD=0. When you leave the SRT both the infinite looping and the onEvent are being stopped.

Of course one could refine it, eg with time-out, algoritm through SRT parameter etc. Also if you just want to wait and not click, just remove the LClick> line etc.

Hope it gives some ideas.

Code: Select all

Let>needle1=%BMP_DIR%\image_3.bmp
Let>needle2=%BMP_DIR%\image_5.bmp
Let>needle3=%BMP_DIR%\image_4.bmp

Gosub>SRT_Wait_Image,needle1,1100,100,1650,400
Wait>0.5
Gosub>SRT_Wait_Image,needle2,1400,100,1600,250
Wait>0.5
Gosub>SRT_Wait_Image,needle3,1050,100,1300,400

MDL>Finished!

SRT>SRT_Wait_Image
    //Waits for needle image by searching hay area given
    //by coordinates and then click center of needle. Call using
    //Gosub>SRT_Wait_Image,needlefile,x0,y0,x1,y1

    //Slow down the onEvent checks and avoid saving screencap to clipboard
    Let>WIN_SLEEP=1
    Let>SCREENCAP_CLIPBOARD=0
    
    //Use local scope
    Let>LOCALVARS=1

    //Get original mouse position to return to at the end 
    GetCursorPos>nXPos,nYPos,csType
    MouseMove>0,0

    //Define Parameters
    CODEBLOCK
        Let>hay=%TEMP_DIR%\screenrect.bmp
        Let>needle=SRT_Wait_Image_VAR_1
        Let>x0=SRT_Wait_Image_VAR_2
        Let>y0=SRT_Wait_Image_VAR_3
        Let>x1=SRT_Wait_Image_VAR_4
        Let>y1=SRT_Wait_Image_VAR_5
        Let>FindFlag=False
    ENDCODEBLOCK

    OnEvent>CUSTOM,MyTriggerSub,DoIT,DoSomething

    Label>Main
        Wait>0.1
        If>FindFlag=True
            Goto>Finish
        Endif
    Goto>Main

    SRT>MyTriggerSub
        Let>LOCALVARS=0
        ScreenCapture>x0,y0,x1,y1,hay
        FindImagePos>needle,hay,0.7,1,XArr,YArr,NumFound,CCOEFF
        If>NumFound>0
            Let>DoIT=TRUE
        Endif
    END>MyTriggerSub

    SRT>DoSomething
        MouseMove>{%XArr_0%+%x0%},{%YArr_0%+%y0%}
        LClick
        OnEvent>CUSTOM,MyTriggerSub,DoIT,
        Let>FindFlag=True
        Let>DoIT=False
    END>DoSomething

    Label>Finish
    
    //Reset parameters and return to original mouse position
    Let>WIN_SLEEP=0
    Let>SCREENCAP_CLIPBOARD=1
    MouseMove>nXPos,nYPos
END>SRT_Wait_Image

[geoffwoz]

Very early days for me with Macro scheduler, so thanks so much for such a detailed and well thought out response (again), and also for the tutorial on custom events, I need to look a bit further into using those in the future.

In terms of scanning a sub-screen area for a matching bitmap, I think for the moment we're left to our own devices with a looping construct and screen capture to a temp file.

Marcus, if you're listening, it would be great to add a feature to allow for blocking scanning of a sub-screen rectangle to avoid having to do this manually, and with the additional overhead of the file I/O.

[hagchr]

I got use for it today so I updated it to factor in Algorithm, Tolerance and if target should be clicked or not. The SRT uses local variable scope so should not interfere with rest of script if accessed through Include> or snippet. It would still be nice to have it included in the normal proper command...

You call using

Code: Select all

Gosub>SRT_Wait_Image,needlefile,Algorithm (CCOEFF/EXACT),Tolerance,x0,y0,x1,y1,click(1/0)

Example

Code: Select all

Gosub>SRT_Wait_Image,%BMP_DIR%\image_1.bmp,CCOEFF,0.7,950,75,1750,500,0

Code: Select all

SRT>SRT_Wait_Image
    //Waits for needle image by searching hay area given
    //by coordinates and then click center of needle. Call using
    //Gosub>SRT_Wait_Image,needlefile,Algoritm (CCOEFF/EXACT),Tolerance,x0,y0,x1,y1,click(1/0)

    //Slow down the onEvent checks and avoid saving screencap to clipboard
    Let>WIN_SLEEP=1
    Let>SCREENCAP_CLIPBOARD=0

    //Use local scope
    Let>LOCALVARS=1

    //Get original mouse position to return to at the end
    GetCursorPos>nXPos,nYPos
    MouseMove>0,0

    //Define Parameters
    CODEBLOCK
        Let>hay=%TEMP_DIR%\screenrect.bmp
        Let>needle=SRT_Wait_Image_VAR_1
        Let>Algoritm=SRT_Wait_Image_VAR_2
        Let>Tolerance=SRT_Wait_Image_VAR_3
        Let>x0=SRT_Wait_Image_VAR_4
        Let>y0=SRT_Wait_Image_VAR_5
        Let>x1=SRT_Wait_Image_VAR_6
        Let>y1=SRT_Wait_Image_VAR_7
        Let>ClickInd=SRT_Wait_Image_VAR_8
        Let>FindFlag=False
    ENDCODEBLOCK

    OnEvent>CUSTOM,MyTriggerSub,DoIT,DoSomething

    Label>Main
        Wait>0.1
        If>FindFlag=True
            Goto>Finish
        Endif
    Goto>Main

    SRT>MyTriggerSub
        Let>LOCALVARS=0
        ScreenCapture>x0,y0,x1,y1,hay
        FindImagePos>needle,hay,Tolerance,1,XArr,YArr,NumFound,Algoritm
        If>NumFound>0
            Let>DoIT=TRUE
        Endif
    END>MyTriggerSub

    SRT>DoSomething
        If>ClickInd=1
            MouseMove>{%XArr_0%+%x0%},{%YArr_0%+%y0%}            
            LClick
            Wait>0.1
        Endif
        OnEvent>CUSTOM,MyTriggerSub,DoIT,
        Let>FindFlag=True
        Let>DoIT=False
    END>DoSomething

    Label>Finish

    //Reset parameters and return to original mouse position
    Let>WIN_SLEEP=0
    Let>SCREENCAP_CLIPBOARD=1
    MouseMove>nXPos,nYPos
END>SRT_Wait_Image