Using Position or Existence of One Image to Select Another Image...

[rjw524]

Hi again!

Most of my work revolves around dynamically generated pages within CRMs and websites, as a result I use a lot of Image Recognition in my scripts.

I am curious if I can use the proximity of one image on a screen to another image on the screen to trigger an action assuming the proximity or distance of the images meet certain conditions?

SITUATION 1:

Basically, I have an image that sometimes shows up on a page. This image says "Go To List".

The issue is this "Go to List" image can occur multiple times on the same page depending on the length of the table to which it's attached. If the table it's attached to is long, it may only appear once. If the tables are short enough the image may appear several times.

The particular "Go to List" image I need to open appears between TWO distinct images. Let's just call them Upper and Lower Image. Depending on how the page generates, these images can be in different places however regardless of where they are Upper image is ALWAYS above Lower image. That condition doesn't change.

Is there any way to tell the script to click the "Go to List" image that is BETWEEN the top and bottom image regardless of where they appear?


SITUATION 2:

Also, is there a way to determine if an image in an Array is a set distance from another image in a different array?

So, for instance, on a dynamic generated page the following table appears:

Fruit 1: Fruit 2:
Apple Orange
Apple
Apple Orange
Apple Orange
Apple

So if wanted the macro to do something for EACH TIME the "Orange" image appears a fixed distance directly across from the "Apple" image (not above or below it) can I script that?

I don't have an example to post because I don't know how to write this script.

I would imagine that MS can do this given it stores the position of the images in an array, but I don't know how I would write something for that.

Thanks so much!

rjw524

[Dorian (MJT support)]

I'm still digesting situation 2, so here's what I have for situation 1. Hopefully it's enough to get you started.

Using the offset, get the top left position of Upper image.
Using the offset, get the bottom right position of Lower image.
Look between those positions for the middle image.

Code: Select all

//Mouse moves not required, but left intact for demonstration purposes
//Top left of Upper image
//Find and Move Mouse Center of 
FindImagePos>%BMP_DIR%\image_1.bmp,SCREEN,0.7,0,UpperXArr,UpperYArr,NumFound,CCOEFF
If>NumFound>0
  MouseMove>UpperXArr_0,UpperYArr_0
Endif
Wait>2


//Mouse moves not required, but left intact for demonstration purposes
//Bottom right of Lower image
//Find and Move Mouse Bottom Right of 
FindImagePos>%BMP_DIR%\image_2.bmp,SCREEN,0.7,4,LowerXArr,LowerYArr,NumFound,CCOEFF
If>NumFound>0
  MouseMove>LowerXArr_0,LowerYArr_0
Endif
Wait>2


//Screencap coordinates between Top/left/upper and bottom/right/lower
ScreenCapture>UpperXArr_0,UpperYArr_0,LowerXArr_0,LowerYArr_0,%TEMP_DIR%\screenrect.bmp

//Search for centre image in that area.
//Find and Left Click Center of 
FindImagePos>%BMP_DIR%\image_3.bmp,%TEMP_DIR%\screenrect.bmp,0.7,1,XArr,YArr,NumFound,CCOEFF
If>NumFound>0
  MouseMove>{%XArr_0%+1507},{%YArr_0%+918}
  LClick
Endif

[rjw524]

I'm still digesting situation 2, so here's what I have for situation 1. Hopefully it's enough to get you started.

Using the offset, get the top left position of Upper image.
Using the offset, get the bottom right position of Lower image.
Look between those positions for the middle image.

Code: Select all

//Mouse moves not required, but left intact for demonstration purposes
//Top left of Upper image
//Find and Move Mouse Center of 
FindImagePos>%BMP_DIR%\image_1.bmp,SCREEN,0.7,0,UpperXArr,UpperYArr,NumFound,CCOEFF
If>NumFound>0
  MouseMove>UpperXArr_0,UpperYArr_0
Endif
Wait>2


//Mouse moves not required, but left intact for demonstration purposes
//Bottom right of Lower image
//Find and Move Mouse Bottom Right of 
FindImagePos>%BMP_DIR%\image_2.bmp,SCREEN,0.7,4,LowerXArr,LowerYArr,NumFound,CCOEFF
If>NumFound>0
  MouseMove>LowerXArr_0,LowerYArr_0
Endif
Wait>2


//Screencap coordinates between Top/left/upper and bottom/right/lower
ScreenCapture>UpperXArr_0,UpperYArr_0,LowerXArr_0,LowerYArr_0,%TEMP_DIR%\screenrect.bmp

//Search for centre image in that area.
//Find and Left Click Center of 
FindImagePos>%BMP_DIR%\image_3.bmp,%TEMP_DIR%\screenrect.bmp,0.7,1,XArr,YArr,NumFound,CCOEFF
If>NumFound>0
  MouseMove>{%XArr_0%+1507},{%YArr_0%+918}
  LClick
Endif

Hi Dorian,

Thanks for the help here...

Ok, it's finding the Upper and Lower images but it doesn't seem to find the third or "in between" image.

I put a message model to show the number of third images it sees and the MDL isn't triggering.

[Dorian (MJT support)]

I'm not sure if this is too simplistic or if I am not understanding situation 2, but if the images are always exactly the same distance apart can't you just sample Apple and Orange together in 1 image?

Failing that, maybe using the method I suggested above to look to the side instead of above and below. You'd have to get the top right and bottom right coordinates, then just add however many pixels you want to scan to the right. This example would be 500.

Code: Select all

//Find and Move Mouse Top Right of 
FindImagePos>%BMP_DIR%\image_5.bmp,SCREEN,0.7,2,TRXArr,TRYArr,NumFound,CCOEFF
FindImagePos>%BMP_DIR%\image_5.bmp,SCREEN,0.7,4,BRXArr,BRYArr,NumFound,CCOEFF

Let>BRX=BRXArr_0+500

ScreenCapture>TRXARR_0,TRYArr_0,BRX,BRYArr_0,%TEMP_DIR%\screenrect.bmp

//Find and Move Mouse Center of 
FindImagePos>%BMP_DIR%\image_6.bmp,%TEMP_DIR%\screenrect.bmp,0.7,1,XArr,YArr,NumFound,CCOEFF
If>NumFound>0
  MouseMove>{%XArr_0%+788},{%YArr_0%+894}
Endif

Of course in both my examples you'll need to run the Image Recognition Wizard as the pixels in my offsets will be different to yours. (MouseMove>{%XArr_0%+788},{%YArr_0%+894} etc)

[Dorian (MJT support)]

To help troubleshoot try adding this line after your ScreenCapture line :

Code: Select all

executefile>%TEMP_DIR%\screenrect.bmp

Then you can at least see if it's looking in the correct place. I tested with this image and it worked perfectly, moving the mouse to the green hexagon.

For the side-by-side I tested using this image.

Once you have it working in principle with these simple shapes and know you're looking in the right place it should just be a matter of fine-tuning your image recognition preference.

[rjw524]

To help troubleshoot try adding this line after your ScreenCapture line :

Code: Select all

executefile>%TEMP_DIR%\screenrect.bmp

Then you can at least see if it's looking in the correct place. I tested with this image and it worked perfectly, moving the mouse to the green hexagon.

For the side-by-side I tested using this image.

Once you have it working in principle with these simple shapes and know you're looking in the right place it should just be a matter of fine-tuning your image recognition preference.

Ok, I ran the "executefile" of the screenrect bitmap and that's working just fine. It also sees the third image. So that's fine as well.

However it doesn't click the third image. Instead it just moves the mouse to the very bottom right of my screen and performs a left click.

I don't think I'm understanding how the following mousemove line is working in this:

MouseMove>{%XArr_0%+1507},{%YArr_0%+918}

To me, this sounds like it's finding the 3rd image, moving to the third image offset by x+1507 and 7+918 and then clicking instead of clicking the image directly.

Am I incorrect?

[Dorian (MJT support)]

Of course in both my examples you'll need to run the Image Recognition Wizard as the pixels in my offsets will be different to yours. (MouseMove>{%XArr_0%+788},{%YArr_0%+894} etc)

I think you may have missed this part of one of my replies. Recreate what I did, using the Image Recognition Wizard.

[rjw524]

Of course in both my examples you'll need to run the Image Recognition Wizard as the pixels in my offsets will be different to yours. (MouseMove>{%XArr_0%+788},{%YArr_0%+894} etc)

I think you may have missed this part of one of my replies. Recreate what I did, using the Image Recognition Wizard.

I didn't miss it, I just don't understand this stuff all well enough yet, I'm so sorry lol. So I guess that's part of my question. None of these images are static or guaranteed to be the same distance from each other. Those coordinates can change every time it's ran so what exactly is that mouse move doing?

I guess I don't know what I'm looking for after running the image recognition myself?

[Dorian (MJT support)]

Those relate to the first and second parameters in your ScreenCapture. I fear I may have misled you by overlooking that, for which I apologize.

If you used the wizard to "search within this rectangle" and it created this code :

Code: Select all

ScreenCapture>1517,419,1802,583,%TEMP_DIR%\screenrect.bmp

The corresponding mousemove would be :

Code: Select all

MouseMove>{%XArr_0%+1517},{%YArr_0%+419}

So in scenario 1 we have this :

Code: Select all

ScreenCapture>UpperXArr_0,UpperYArr_0,LowerXArr_0,LowerYArr_0,%TEMP_DIR%\screenrect.bmp

So what you'd need is this (tested and working) :

Code: Select all

MouseMove>{%XArr_0%+%UpperXArr_0%},{%YArr_0%+%UpperYArr_0%}

[Dorian (MJT support)]

Those coordinates can change every time it's ran so what exactly is that mouse move doing?

The script will :

1. Find the top left corner position of the top image.
2. Find the bottom right corner position of the bottom image.

The middle image will be within a rectangle we create using those two coordinates.

So,

3. Screencapture that rectangle.
4. Look within that rectangle for the middle image.

The position should not matter, as long as it's within that rectangle.

Apologies for the confusion, i was experimenting with it myself.

[rjw524]

Those coordinates can change every time it's ran so what exactly is that mouse move doing?

MouseMove>{%XArr_0%+%UpperXArr_0%},{%YArr_0%+%UpperYArr_0%}

That worked perfectly for the first scenario! Thanks for explaining it so thoroughly! It really helps!

[/quote]
Apologies for the confusion, i was experimenting with it myself.
[/quote]

No apologies necessary in the slightest. As a non-programmer being able to get this kind of help is golden.

[rjw524]

Those coordinates can change every time it's ran so what exactly is that mouse move doing?

MouseMove>{%XArr_0%+%UpperXArr_0%},{%YArr_0%+%UpperYArr_0%}

That worked perfectly for the first scenario! Thanks for explaining it so thoroughly! It really helps!

Apologies for the confusion, i was experimenting with it myself.

No apologies necessary in the slightest. As a non-programmer being able to get this kind of help is SO appreciated.

[rjw524]

MouseMove>{%XArr_0%+%UpperXArr_0%},{%YArr_0%+%UpperYArr_0%}

That worked perfectly for the first scenario! Thanks for explaining it so thoroughly! It really helps!

Apologies for the confusion, i was experimenting with it myself.

No apologies necessary in the slightest. As a non-programmer being able to get this kind of help is SO appreciated.
[/quote]

[rjw524]

So, Scenario 2, is a little more complicated (to me anyway) because it needs to move through an array.

I put in your code and ran image recognition to get my own coordinates but it's not moving through the array.

It goes to the first example but that's it.

With Scenario 2 the trick is sometimes the 2nd image appears next to 1st image and sometimes it doesn't.

Here's the code I put in place.

Code: Select all

//Set IGNORESPACES to 1 to force script interpreter to ignore spaces.
//If using IGNORESPACES quote strings in {" ... "}
//Let>IGNORESPACES=1

SetFocus>Microsoft​ Edge*

Wait 1.0

//Find and Move Mouse Top Right of 
FindImagePos>%BMP_DIR%\image_1.bmp,SCREEN,150,2,TRXArr,TRYArr,EditFound,EXACT
FindImagePos>%BMP_DIR%\image_1.bmp,SCREEN,150,4,BRXArr,BRYArr,EditFound,EXACT
Let>k=0
Repeat>k
Let>BRX=BRXArr_0+635
ScreenCapture>{%TRXArr_0%-50},{%TRYArr_0%-0},BRX,BRYArr_0,%TEMP_DIR%\screenrect.bmp

wait 1.0

//Find and Move Mouse Center of 
FindImagePos>%BMP_DIR%\image_2.bmp,%TEMP_DIR%\screenrect.bmp,50,1,XArr,YArr,CheckFound,CCOEFF
If>NumFound>0
  MouseMove>{%XArr_0%+%TRXArr_0%},{%YArr_0%+%TRYArr_0%}
Endif

Wait 5.0
  Let>k=k+1
  Until>k={%EditFound%}

I also sent you a PM, Dorian.

Thanks!

[Dorian (MJT support)]

Got it.

I notice in your last snippet you're working in Edge.

If this is Edge, why not just use the v15 Edge functions to click these buttons?

Learning those, and the Chrome ones (if you haven't already), would open up a whole new world for you.