Image Recording with Mouse Click

Technical support and scripting issues

Moderators: Dorian (MJT support), JRL

Post Reply
ocnuybear
Pro Scripter
Posts: 100
Joined: Sun Jul 15, 2018 5:14 pm

Image Recording with Mouse Click

Post by ocnuybear » Tue Sep 20, 2022 8:18 am

Hallo everyone, I want to record my clicks on any desktop app, winform or chrome, etc, using MS ScreenCapture, which is easy, but when moving the mouse over an element or button in webpage or winform, it triggers mouse over events thus changing the picture before I can capture it for playback later using MS FindImagePos.

My idea is to create a transparent Form over the whole screen and capture a small predifened area around the cursor, close the transparent form for the mouse click/mouse over to perform, create the Form again before next action, etc.

This is the code I started to used to create this transparent FORM:

Code: Select all

Dialog>Dialog1
   Label=g,4,4
EndDialog>Dialog1
Let>pxw=90
Let>pxh=60
Let>X1Offset=500
Let>Y1Offset=500
Let>Left1=X1Offset
Let>Top1=Y1Offset
Let>Right1=pxw+X1Offset
Let>Bottom1=pxh+Y1Offset
Let>height1=pxh+100
Let>width1=pxw+100
Let>Y1Offset=Y1Offset-5
Let>X1Offset=X1Offset-10
SetDialogProperty>Dialog1,,Position,poDesigned
SetDialogProperty>Dialog1,,Left,X1Offset
SetDialogProperty>Dialog1,,Top,Y1Offset
SetDialogProperty>Dialog1,,clientheight,height1
SetDialogProperty>Dialog1,,clientwidth,width1
SetDialogProperty>Dialog1,,Color,255
//Set Text Color (color number 234567 is a shade of green)
SetDialogObjectFont>Dialog1,msLabel1,WebDings,120,1,234567
//Removes borders
LibFunc>user32,SetWindowLongA,sres,dialog1.handle,-16,1073741824
//Open the dialog (normal appearance)
Show>Dialog1
//constants
  Let>GWL_EXSTYLE=-20
  Let>WS_EX_LAYERED=524288
  Let>LWA_COLORKEY=1
//get style attributes of dialog1 window
  LibFunc>user32,GetWindowLongA,attribs,dialog1.handle,GWL_EXSTYLE
  Let>attribs={%attribs% OR %WS_EX_LAYERED%}
//make selected color transparent (color number 234567)
  LibFunc>user32,SetWindowLongA,swl,dialog1.handle,GWL_EXSTYLE,attribs
  LibFunc>user32,SetLayeredWindowAttributes,res,dialog1.handle,234567,0,LWA_COLORKEY
//Make the button disappear
  LibFunc>User32,ShowWindow,SWres,dialog1.msbutton1.handle,0
//Show the dialog without text for a couple of seconds
  CloseDialog>dialog1
  ResetDialogAction>dialog1
Label>Start
  Show>dialog1
Goto>Start
There is code in there to for the offset as well as it's size.
The two biggest chanllenges I have is to:
1. Change the g into a 16:9 ratio, but will need a different object that can be sized seperately between Heigt & Width to get this ratio.

2. Played around using different objects like panels & memos that can be set like that, but cannot get anything to go inivisble as soon as the Dialog Designer places some extra code so why does this work?

Code: Select all

Dialog>Dialog1
   Label=g,4,4
EndDialog>Dialog1
while changing it with Dialog Designer to

Code: Select all

Dialog>Dialog1
object Dialog1: TForm
  Left = 479
  Top = 205
  HelpContext = 5000
  BorderIcons = [biSystemMenu]
  Caption = 'CustomDialog'
  ClientHeight = 203
  ClientWidth = 586
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -14
  Font.Name = 'MS Sans Serif'
  Font.Style = []
  OldCreateOrder = True
  ShowHint = True
  OnTaskBar = False
  PixelsPerInch = 120
  TextHeight = 16
  object msLabel1: TLabel
    Left = 4
    Top = 4
    Width = 8
    Height = 16
    Caption = 'g'
  end
end
EndDialog>Dialog1
The name msLabel1 did not change, but does not go invisible using the code above?

User avatar
Dorian (MJT support)
Automation Wizard
Posts: 1348
Joined: Sun Nov 03, 2002 3:19 am
Contact:

Re: Image Recording with Mouse Click

Post by Dorian (MJT support) » Tue Sep 20, 2022 9:46 am

Does this absolutely need to be done with a Dialog? It seems an OnEvent would do the job.

This does what I think you're trying to do.

Code: Select all

OnEvent>key_down,VK1,0,Clicked

Label>Loop
Wait>0.01
Goto>Loop

SRT>Clicked
GetCursorPos>NowX,NowY
Let>X1=NowX-100
Let>Y1=NowY-100

//Remove negative values
If>X1<0
  Let>X1=0
Endif
If>Y1<0
  Let>Y1=0
Endif

Let>X2=NowX+100
Let>Y2=NowY+100

Year>TheYear
Month>TheMonth
Day>theDay
Hour>TheHour
Min>TheMin
Sec>TheSeconds

ScreenCapture>X1,Y1,X2,Y2,d:\Capture-%TheYear%-%TheMonth%-%theDay%-%TheHour%-%TheMin%-%TheSeconds%.bmp
END>Clicked

One thing to be careful of is that if they click twice in the same second, the first capture would be overwritten. If that's likely and undesired then you'd need to add something in to change the file name if one for that exact second already exists.
Yes, we have a Custom Scripting Service. Message me or go here

ocnuybear
Pro Scripter
Posts: 100
Joined: Sun Jul 15, 2018 5:14 pm

Re: Image Recording with Mouse Click

Post by ocnuybear » Tue Sep 20, 2022 10:37 am

Hi Dorian,

Yes the script you provided is on the right track, the only reason for me to use an invisible dialog is to prevent any mouse over/hover/click action to change the area of importance's picture, for example www,google.com on the right hand side the Gmail link will display Gmail, but hover the cursor over it and it changes to Gmail with a line under neath it, I need to capture this Gmail image it was before hovered/clicked to use FindImagePos to find it again when playing back the script and if a dialog or other window is on top of the browser window it, the browser will not detect the mouse over it.
Last edited by ocnuybear on Tue Sep 20, 2022 12:16 pm, edited 2 times in total.

User avatar
Dorian (MJT support)
Automation Wizard
Posts: 1348
Joined: Sun Nov 03, 2002 3:19 am
Contact:

Re: Image Recording with Mouse Click

Post by Dorian (MJT support) » Tue Sep 20, 2022 10:56 am

Aah, I see. I have very little experience with transparent Dialogs, but I know there are a few people here who have used them and may be able to offer some of their insights. Let's see if they chime in. If they don't I'll refer the question to Marcus.
Yes, we have a Custom Scripting Service. Message me or go here

ocnuybear
Pro Scripter
Posts: 100
Joined: Sun Jul 15, 2018 5:14 pm

Re: Image Recording with Mouse Click

Post by ocnuybear » Tue Sep 20, 2022 12:03 pm

For now I have scavenged the rubber band selection script by JRL that instead of making the dialog invisible makes it semi see through viewtopic.php?t=4156

Code: Select all

OnEvent>KEY_DOWN,VK27,0,Stop
OnEvent>KEY_DOWN,VK90,0,Stop
Let>SM_CXVIRTUALSCREEN=78
Let>SM_CYVIRTUALSCREEN=79
Let>SM_XVIRTUALSCREEN=76
Let>SM_YVIRTUALSCREEN=77
LibFunc>User32,GetSystemMetrics,VScrWidth,SM_CXVIRTUALSCREEN
LibFunc>User32,GetSystemMetrics,VScrHeight,SM_CYVIRTUALSCREEN
LibFunc>User32,GetSystemMetrics,VScrLeft,SM_XVIRTUALSCREEN
LibFunc>User32,GetSystemMetrics,VScrTop,SM_YVIRTUALSCREEN
//Minimal dialog specifications no border and set alphablend to 100
Dialog>RubberBandSelectionDialog
object RubberBandSelectionDialog: TForm
  BorderStyle = bsNone
  AlphaBlend = True
  AlphaBlendValue = 100
end
EndDialog>RubberBandSelectionDialog
//Set the dialog window to cover the virtual screen area
Let>WIN_USEHANDLE=1
  ResizeWindow>RubberBandSelectionDialog.handle,VScrWidth,VScrHeight
  MoveWindow>RubberBandSelectionDialog.handle,VScrLeft,VScrTop
Let>WIN_USEHANDLE=0
//Display the semi transparent dialog
Label>Start
  Show>RubberBandSelectionDialog
Goto>Start
SRT>Stop
  Exit
END>Stop
Firstly the new issue that came up is the click creates several images, due to the nature of the mouse's mechanical click. In electronics you would add a digital debouncer.

User avatar
JRL
Automation Wizard
Posts: 3497
Joined: Mon Jan 10, 2005 6:22 pm
Location: Iowa

Re: Image Recording with Mouse Click

Post by JRL » Wed Sep 21, 2022 2:19 pm

Sorry I don't have time to write this for you but I think you're on the right track. What I would do is:

- Invoke the semi-transparent dialog and click on the dialog.
- Capture the click location and directly after move the mouse to a safe location.
- Directly after that close the semi-transparent dialog and use the captured mouse coordinates to acquire your image.

Also, you can make the dialog practically invisible yet still mask the screen by setting AlphaBlendValue = 1. Visually the dialog is invisible but I've never tested to see how much affect that has on the actual pixel colors of a captured image, assuming you'd want to attempt to capture images without closing the dialog.

ocnuybear
Pro Scripter
Posts: 100
Joined: Sun Jul 15, 2018 5:14 pm

Re: Image Recording with Mouse Click

Post by ocnuybear » Wed Sep 21, 2022 5:10 pm

JRL wrote:
Wed Sep 21, 2022 2:19 pm
Sorry I don't have time to write this for you but I think you're on the right track. What I would do is:

- Invoke the semi-transparent dialog and click on the dialog.
- Capture the click location and directly after move the mouse to a safe location.
- Directly after that close the semi-transparent dialog and use the captured mouse coordinates to acquire your image.

Also, you can make the dialog practically invisible yet still mask the screen by setting AlphaBlendValue = 1. Visually the dialog is invisible but I've never tested to see how much affect that has on the actual pixel colors of a captured image, assuming you'd want to attempt to capture images without closing the dialog.
No worries, I just needed some push in the right direction, thank you for using your old script & some advice :D

ocnuybear
Pro Scripter
Posts: 100
Joined: Sun Jul 15, 2018 5:14 pm

Re: Image Recording with Mouse Click

Post by ocnuybear » Wed Sep 21, 2022 7:01 pm

I have a working script thanks to everybody that replied here, start chrome, then start the script, it will create a C:\Capture directory and will capture everything you click on, idea is to make another script using MS FindImagePos to find the position and click the images you have captured, should work for any Windows Desktop app in theory. Use Escape to end the macro and don't put in any breakpoints as OnEvent will do strange things with MS, not sure why.

Code: Select all

OnEvent>KEY_DOWN,VK27,0,Stop
OnEvent>key_down,VK1,0,Clicked
Dialog>Dialog1
object Dialog1: TForm
  AlphaBlend = True
  AlphaBlendValue = 1
  Caption = 'CustomDialog'
  BorderStyle = bsNone
end
EndDialog>Dialog1
//Creates Capture Dir if not exists
IfNotDirExists>C:\Capture
  CreateDir>C:\Capture
ENDIF
//Please set your screen size here
Let>pxw=1918
Let>pxh=1075
Let>X1Offset=1
Let>Y1Offset=1
Let>Left1=X1Offset
Let>Top1=Y1Offset
Let>Right1=pxw+X1Offset
Let>Bottom1=pxh+Y1Offset
Let>height1=pxh+100
Let>width1=pxw+100
Let>Y1Offset=Y1Offset-5
Let>X1Offset=X1Offset-10
SetDialogProperty>Dialog1,,Position,poDesigned
SetDialogProperty>Dialog1,,Left,X1Offset
SetDialogProperty>Dialog1,,Top,Y1Offset
SetDialogProperty>Dialog1,,clientheight,height1
SetDialogProperty>Dialog1,,clientwidth,width1
//Loop through a list of windows and activates Chrome window - please start this before running script
GetWindowList>winlist
Separate>winlist,CRLF,windows
Let>k=1
Repeat>k
  Let>this=windows_%k%
  Position>Google Chrome,this,1,nPos,TRUE
  IF>nPos>0
    WindowAction>1,this
    Let>k=windows_count
  ENDIF
  Let>k=k+1
Until>k>windows_count
IF>nPos=0
  MessageModal>Please start Chrome before running this script!
  Exit
ENDIF
Label>Loop
  Wait>0.01
  Show>Dialog1
Goto>Loop
SRT>Clicked
  IFNot>nPos=0
    GetCursorPos>NowX,NowY
    Let>X1=NowX-100
    Let>Y1=NowY-20
    //Remove negative values
    If>X1<0
      Let>X1=0
    Endif
    If>Y1<0
      Let>Y1=0
    Endif
    Let>X2=NowX+100
    Let>Y2=NowY+20
    Year>TheYear
    Month>TheMonth
    Day>theDay
    Hour>TheHour
    Min>TheMin
    Sec>TheSeconds
    ScreenCapture>X1,Y1,X2,Y2,C:\Capture\Snapshot%TheYear%%TheMonth%%theDay%%TheHour%%TheMin%%TheSeconds%.bmp
    Wait>0.1
    CloseDialog>Dialog1
    Wait>0.1
    MouseMove>NowX,NowY
    LClick
    Wait>1
    //ExecuteFile>c:\Capture
  Endif
END>Clicked
SRT>Stop
  Exit
END>Stop

Post Reply
Sign up to our newsletter for free automation tips, tricks & discounts