March 12, 2020

Sending Keystrokes to Elements in Chrome and Edge Using the New Chrome/Edge Functions

Filed under: Automation,Web/Tech — Marcus Tettmar @ 12:56 pm

It may not be immediately obvious that as well as setting the value of elements with ChromeSetElementValue and EdgeSetElementValue you can also send non-character keystrokes, such as Enter, or Page Down.

To do this, use the key codes for Edge and Chrome listed here.

For example:

ChromeSetElementValue>session_id,message_elements_1,First line
//Press enter on the element ... 
ChromeSetElementValue>session_id,message_elements_1,\uE006
ChromeSetElementValue>session_id,message_elements_1,Second line

Note that when sending keystrokes only one can be sent at a time.

Recently someone needed to scroll down inside a div, in order to force the page to fetch more data. Issuing a Page Down on the div element did the trick:

ChromeSetElementValue>session_id,div_elements_1,\uE00F

So, ChromeSetElementValue sends keys as well as sets values – perhaps it should have been called ChromeSendKeysToElement and just maybe we’ll add that as a mapping if it helps.

Of course you don’t have to use the new Chrome/Edge functions – you can still use UI methods and use SetFocus, SendText and/or Image Recognition to locate page elements.

March 6, 2020

Working with Frames in Chrome and Edge

Filed under: Automation,Scripting,Web/Tech — Marcus Tettmar @ 1:28 pm

The new Chrome and Edge functions in Macro Scheduler 15 make it possible to locate frames and iframes and then manipulate elements within them.

Switching Frames

The ChromeSwitchFrame and EdgeSwitchFrame functions allow you to specify a frame or iframe element, by index or element reference, to switch the “browsing context” to.

What this means is that any subsequent interactions will take place against elements within that frame. So a subsequent ChromeFindElements call will attempt to locate the specific element within the frame, rather than the parent page, and ChromeElementAction will act against the given element within that frame.

//Find the frame using xpath
ChromeFindElements>sessionID,xpath,//iframe[@src='/contact-form/formpage.html'],elements

//switch browsing context to this frame
ChromeSwitchFrame>session_id,element,elements_1,res

If xpath doesn’t mean anything to you see my recent post Using Macro Scheduler 15’s Chrome Functions which includes an explanation of using Chrome’s Developer Tools to identify elements.

At some point you may need to switch the browsing context back to the parent frame. To do this you call ChromeSwitchFrame/EdgeSwitchFrame again with a null index. Subsequent calls to the Chrome/Edge functions will then act against the parent frame.

ChromeSwitchFrame>session_id,index,null,res

Traversing Frames

Since each time you switch frames you change the context to that frame, calling ChromeSwitchFrame or EdgeSwitchFrame again (on a valid frame element within) will switch the context down another level.

//Find the frame using xpath
ChromeFindElements>sessionID,xpath,//iframe[@src='/contact-form/formpage.html'],elements
ChromeSwitchFrame>session_id,element,elements_1,res

//Switch to the next frame down *within the current frame*
ChromeSwitchFrame>session_id,index,0,res

Differences between Edge and Chrome

The Edge and Chrome functions work in the same way and are almost identical. There’s one major difference when it comes to switching frames. When specifying an element (rather than index) ChromeSwitchFrame requires the element ID, whereas EdgeSwitchFrame requires the full element object. As well as an array of element IDs, EdgeFindElements returns a second array of the objects:

EdgeFindElements>sessionID,xpath,//iframe[@src='/contact-form/formpage.html'],FrameElements

This returns two arrays prefixed with the name passed as the return var (TheElements): FrameElements_1 … FrameElements_n and FrameElements_objects_1 … FrameElements_objects_n. For EdgeSwitchFrame use the second _objects array:

//switch to the frame
EdgeSwitchFrame>session_id,element,FrameElements_objects_1,res

More Help

View the Chrome Functions and Edge Functions topics in the Macro Scheduler Manual.

Full Example:

Let>CHROMEDRIVER_EXE=c:\chromedriver.exe

//start a Chrome session
ChromeStart>session_id

//navigate to google.com
ChromeNavigate>session_id,url,https://www.mjtnet.com/contact.htm

//Find the frame
ChromeFindElements>sessionID,xpath,//iframe[@src='/contact-form/formpage.html'],elements

//switch to the first frame (the one with the fields)
ChromeSwitchFrame>session_id,element,elements_1,res

//now anything we do is inside that frame so we should be able to get the name field and enter something
ChromeFindElements>session_id,id,name,inputs
ChromeSetElementValue>session_id,inputs_1,john doe

//Switch context back to parent
ChromeSwitchFrame>session_id,index,null,res

February 27, 2020

Using Macro Scheduler 15’s Chrome Automation Functions

Filed under: Automation,Scripting,Web/Tech — Marcus Tettmar @ 6:48 pm

Macro Scheduler 15 now has native Chrome and Edge functions. You may be wondering how they work. There are a couple of sample scripts included with v15 but if you’re not sure how to get things set up or how to identify elements in the page this article is for you.

I wanted an example that wasn’t too involved but juicy enough to offer a few hurdles and demonstrate some features like xpath. I’ve set upon using Yahoo Finance to pull out the value of a stock symbol. Bear in mind that websites have a habit of changing so I can’t guarantee that the code in this article will work 6 months from now. But that shouldn’t matter much – it’s the principles of how to do this that matters most.

Before we begin make sure you are running Macro Scheduler 15.0.06 or above.

ChromeDriver

Before we can use the new Chrome functions we need to download the win32 version of ChromeDriver from: https://chromedriver.chromium.org

The zip file contains a single file: chromedriver.exe. Save this somewhere.

We need to tell the script where chromedriver.exe is. In my examples below I’ve got it in the root of drive C. So the first line of my script needs to be:

Let>CHROMEDRIVER_EXE=c:\chromedriver.exe

Start a Chrome Session

Now, we can start a Chrome session. To do this we use the ChromeStart function and give it a variable name that we want the session ID stored in. I’m using session_id. We’ll need this for the rest of the commands:

ChromeStart>session_id

Run your script and if your script is correctly pointing at ChromeDriver.exe you should see an empty Chrome window appear.

Navigating

To navigate to Yahoo add the ChromeNavigate command, passing the session_id to it:

ChromeNavigate>session_id,url,https://finance.yahoo.com/

Close the Chrome browser window (you don’t have to but running your script is going to start another) and run your script again.  This time you should see it navigate to Yahoo and show something like this:

Using Developer Tools

So the first thing we need to do is get past that “I agree” button. This is where Chrome’s Developer Tools comes in to help us find the element and the attributes we can use to identify it.

Right click on the “I agree” button and select “Inspect” at the bottom of the popup menu.  Chrome’s Developer Tools should appear. Usually at the bottom, but if you’ve used it before you may have changed the layout. It should appear in ‘Elements’ view and the highlighted element should be the “I agree” button:

Sometimes, I’ve found that the first time I select “Inspect”, if Developer Tools wasn’t already open, the element I want isn’t the highlighted one. Usually right clicking on the element and selecting Inspect a second time does the trick.

Another way to identify the element once Developer Tools is already open is to click the element selector button which you see at top left in the screenshot above. Then move the mouse to the “I agree” button and click.

Once you’ve got the right element selected look for an attribute which we could use to identify it:

<button type="submit" class="btn primary" name="agree" value="agree">I agree</button>

Ideally we want something unique.That isn’t always possible and I’ll talk about what we can do in those cases later. Here we’re in luck. The element has a name – “agree”. We can use this to get a reference to the element so that we can then ‘click’ it:

ChromeFindElements>session_id,name,agree,agree_elements

Notice how we’re passing the session ID again (we want to make sure we’re referring to the Chrome window we started). The second parameter is the “Strategy” we want to use to identify the element. ‘Name’ is an available strategy – find the element by name. The third parameter is the strategy value. For ‘Name’ it’s the name itself which in this case is “agree”. Finally we give the command a variable name we’d like to store the located element IDs in.

Valid ‘strategies’ you can use are:

id, name, class name, css selector, link text, partial link text, tag name, xpath

Clicking on the Element

ChromeFindElements returns an array of matching elements. In this case the array should have only one item because there’s only one matching element. So the item we want is the first one: agree_elements_1.

Let’s use that to click on the button using the ChromeElementAction command:

ChromeElementAction>session_id,agree_elements_1,click

The script so far:

Let>CHROMEDRIVER_EXE=c:\chromedriver.exe
ChromeStart>session_id
ChromeNavigate>session_id,url,https://finance.yahoo.com/
ChromeFindElements>session_id,name,agree,agree_elements
ChromeElementAction>session_id,agree_elements_1,click

Close your browser window and re-run the script. Hopefully you’ll sail right through that I agree box.

ChromeDriver should wait for events to complete and pages to load before firing new events, but if you have any issues you may need to slow things down a touch. E.g. stick a Wait>1 before the click.

Sending the Stock Symbol

We now need to find that input box, enter something and submit it. Using the same ‘Identify’ technique as described above you should find that the input element has its “id” attribute set to “yfin-usr-qry”:

So we’ll use that:

ChromeFindElements>session_id,id,yfin-usr-qry,input_elements

I’ve used input_elements as my array name for the found elements. We now want to enter some text. Let’s search for Microsoft which is stock symbol “MSFT”. To do this we’ll use the ChromeSetElementValue command. There’s only one matching element again so as above we use the first match input_elements_1:

ChromeSetElementValue>session_id,input_elements_1,MSFT
Wait>1

Notice this time I have put a wait afterwards. The next step is to ‘submit’ the field. I found that submitting it without the wait didn’t allow enough time for the entry to ‘take’. There could be some background processing going on which needs time to finish. Remember, as with most automation, we’re automating something that was designed for a user, and Robots run faster than users, so sometimes need slowing down!

We will now use the ChromeElementAction command again, this time with the submit action:

ChromeElementAction>session_id,input_elements_1,submit

Instead of doing this we could have found the search button and issued a ‘click’ on that. But that would require another ChromeFindElements which in this case is unnecessary. We can ‘submit’ the input directly.

Extracting The Stock Value

Rerun the script and you should end up looking at the Microsoft stock info which we want to extract.

Tip: rerunning the script from the top every time isn’t ideal. We’ve got the Chrome window open so the session is still active. So what you could do if you just want to run and test a few commands is grab the session ID from the watch list and assign it with a Let statement placed just before the commands you want to test. E.g:

Let>session_id=b6976cfec0326dcce35ad3674c3ed90e
ChromeFindElements>session_id,id,yfin-usr-qry,input_elements
ChromeSetElementValue>session_id,input_elements_1,MSFT

This is where things get a little more interesting. Use the inspect element tool to locate the element as above. You’ll see it’s a span element. You might also notice the class keeps changing. This site is very dynamic, updating as the stock value changes. In this instance we aren’t lucky enough to have an id, or a name. But notice those custom ‘data-reactid’ attributes.  You’ll see that they all have different values, and the one we want is “34”.  Let’s see if we can use that.

Now, ‘data-reactid’ is a custom attribute. It’s not standard html, and there’s no ‘strategy’ called that. So we’re going to use something called ‘xpath’. xpath is incredibly powerful and can help us find pretty much anything. There’s a great tutorial on xpath here.

We need to find a ‘span’ element with attribute data-reactid which is set to “34”. In xpath we can do this with:

//span[@data-reactid='34']

The // means find nodes regardless of where they are – we don’t need to provide a path. The @ symbol is used to specify an attribute. And we’re giving it a value. We’re saying find a span element with attribute data-reactid set to value “34” and we don’t care where it is.

So we can use this in the ChromeFindElements command as follows:

ChromeFindElements>session_id,xpath,//span[@data-reactid='34'],info_elements

Now, before we continue we should confirm whether or not this is the ONLY span tag with this attribute and value. We could do that either by searching the source using Chrome’s element inspector, or, I think easier, just debug the code to see what we get back. Stick a breakpoint just after this line, run the script and then look at the watch list. You’ll see there are two:

Now, we’re going to need to extract them to see which is the right one. We’re going to use the ChromeGetElementData command:

ChromeGetElementData>session_id,info_elements_1,text,stockValue

If you step through this command and look at the value of stockValue in the watch list, you’ll notice that info_elements_1 is not the one we want. We want the second one:

ChromeGetElementData>session_id,info_elements_2,text,stockValue
MessageModal>Value is %stockValue%

So there we have it. We’ve started Chrome, navigated to Yahoo Finance, clicked ‘I agree’, searched for MSFT and extracted the stock value.  We might want to close the Chrome window at the end:

ChromeQuit>session_id

Here’s the full script:

Let>CHROMEDRIVER_EXE=c:\chromedriver.exe

ChromeStart>session_id
ChromeNavigate>session_id,url,https://finance.yahoo.com/

ChromeFindElements>session_id,name,agree,agree_elements
ChromeElementAction>session_id,agree_elements_1,click

ChromeFindElements>session_id,id,yfin-usr-qry,input_elements
ChromeSetElementValue>session_id,input_elements_1,MSFT
Wait>1

ChromeElementAction>session_id,input_elements_1,submit
Wait>1

ChromeFindElements>session_id,xpath,//span[@data-reactid='34'],info_elements
ChromeGetElementData>session_id,info_elements_2,text,stockValue
MessageModal>stockValue

ChromeQuit>session_id

Using Edge instead of Chrome?

The Edge commands work in exactly the same way as Chrome. And finding elements in Edge is done in exactly the same way. So you should be able to adapt the above code to work with Edge very easily. The main difference is that Edge is a bit more picky when it comes to installing the correct version of Edge Driver.  And if you have an older ‘legacy’ version of Edge you will need to install Microsoft Web Driver instead of MS Edge Driver.  The manual explains how.

Conclusion

The Chrome and Edge functions are incredibly powerful and finding elements using Chrome’s, or Edge’s, Developers Tools is quick and easy. There’s also a lot more that you can do, such as grab the entire source of the page. I hope this has been useful. Let me know in the comments below how you are using these functions.

You’ll find more info in the manual. See Chrome Functions and Edge Functions.

If you have any questions find me on the forums or drop us a line.

July 20, 2018

Cross Language 128 and 256 Bit AES Encryption

Filed under: Scripting,Web/Tech — Marcus Tettmar @ 10:47 am

With Macro Scheduler 14.4.10 we have added new 128 and 256 bit AES Encryption methods which interoperate well with other languages.

Cross-platform/cross-language encryption/decryption can be tricky. Different text encodings, different padding formats and understanding how different algorithms derive keys and initialisation vectors can make encrypting and decrypting between one language and another a bit of a pain!

Macro Scheduler’s AES function originally worked only on Unicode strings as this is the standard string format in Macro Scheduler.  Having had some issues trying to make this work between NodeJS and PHP we decided to create some new UTF8 based AES options (available in Macro Scheduler 14.4.10) which we have verified work well with PHP/OpenSSL and NodeJS.  

These new implementations provide 128 and 256 bit AES encryption.  They use CBC chaining method and use a SHA256 password. If using AES 128 the SHA256 password is truncated to the 32 byte key length.  By default the initialisation vector is set to “0000000000000000” but can be set to whatever you want (but must be 16 characters long). Padding is PKCS#5.

Here is an example of AES_256:

Let>AES_ALG=AES_256_CBC
AESEncrypt>hello world,mypassword,ENCRYPT,result
AESEncrypt>result,mypassword,DECRYPT,original


Here is a compatible PHP example:

data = "hello world";

$method = 'AES-256-CBC';

// simple password hash
$password = 'mypassword';
$key = hex2bin(substr(hash('sha256', $password),0,64));

echo "Method: " . $method . "\n";
$encrypted = encrypt($data, $key, $method);
echo "Encrypted: ". $encrypted . "\n";
$decrypted = decrypt($encrypted, $key, $method);
echo "Decrypted: ". $decrypted . "\n"; // plain text

function encrypt(string $data, string $key, string $method): string
{
  $iv = "0000000000000000";
  $encrypted = openssl_encrypt($data, $method, $key, OPENSSL_RAW_DATA, $iv);
  $encrypted = base64_encode($encrypted);
  return $encrypted;
}

function decrypt(string $data, string $key, string $method): string
{
  $data = base64_decode($data);
  $iv = "0000000000000000";
  $data = openssl_decrypt($data, $method, $key, OPENSSL_RAW_DATA,$iv);
  return $data;
}

Try this PHP Code Here

And here’s the Javascript equivalent:

let data = 'hello world';
let password = 'mypassword';
let iv = '0000000000000000';

let password_hash = crypto.createHash('sha256').update(password,'utf8').digest('hex');

let key = hex2bin(password_hash);
password_hash = Buffer.alloc(32,key,"binary");

let cipher = crypto.createCipheriv('aes-256-cbc', password_hash, iv);

let encryptedData = cipher.update(data, 'utf8', 'base64') + cipher.final('base64');

console.log('Base64 Encrypted:', encryptedData);

let decipher = crypto.createDecipheriv('aes-256-cbc', password_hash, iv);

let decryptedText = decipher.update(encryptedData, 'base64', 'utf8') + decipher.final('utf8');

console.log('Decrypted Text:', decryptedText)

function hex2bin(hex)
{
  var bytes = [], str;

  for(var i=0; i< hex.length-1; i+=2)
    bytes.push(parseInt(hex.substr(i, 2), 16));

  return String.fromCharCode.apply(String, bytes);
}

Try this Javascript code here.

You can try the Javscript and PHP examples at repl.it:

https://repl.it/@MarcusTettmar/phpaes256cbc1
https://repl.it/@MarcusTettmar/jsaes267cbc1

April 2, 2018

Escaping File Paths in HTTP Requests

Filed under: Automation,Web/Tech — Marcus Tettmar @ 5:48 pm

If you ever need to send a path as a URL parameter in an HTTP request, be sure to “escape” it to avoid the path delimiters (“\” symbols) being seen as delimiters in the URL path.

E.g. let’s say you need to send a filename as a parameter in a GET request, like this:

https://someserver.com/resource?filename=c:\my files\subfolder\filename.txt

If we fail to escape those “\” characters we are likely to get a server error or a 404 not found error at best. The space character in the above example is also likely to upset things!

We can solve this by using VBScript’s Escape function to “URL Encode” the string. What this does is replace the special characters with a special code made of a % symbol and two hexadecimal digits. E.g. a space character is replaced with %20.

So, we should use the Escape function something like this:

Let>strFilename=c:\my files\subfolder\filename.txt
VBEval>Escape("%strFilename%"),strFilename

Then we can safely do:

HTTPRequest>https://someserver.com/resource?filename=%strFilename%,,GET,,strResponse

If you were to paste the URL into Google Chrome or IE you will find that behind the scenes they apply this encoding for you.

Tip: A great way to test HTTP requests is to use http://webhook.site – this gives you a special URL you can send your data to (in place of your real web service) and for each request you make it will show you what it received. Try manually sending a filename like the one above using your web browser and you’ll see those % codes being added for you.

November 24, 2015

Finding HTML Attributes For Automating Web Sites

Filed under: Automation,Scripting,Web/Tech — Marcus Tettmar @ 1:00 pm

If you’re new to automating IE/websites with WebRecorder or the native Macro Scheduler IE functions you may be wondering how to determine which elements and attributes to use.

In this video I demonstrate how to use IE’s F12 key to invoke Developer Tools and use that to quickly find the elements we’re interested in and the attributes we need to use:

(You might want to click on the video toolbar to select a larger resolution size, view full screen or view on YouTube so that you can see the code).

January 24, 2012

WebRecorder 3.0 – Faster, More Reliable, More Functions

Filed under: Automation,Web/Tech — Marcus Tettmar @ 11:04 pm

We have today released WebRecorder 3.0.

WebRecorder 3.0 - Faster more reliable internet macros

This is a major rewrite with a better recording engine capable of recording more tags and producing leaner, cleaner, code and a faster and more reliable runtime.

It also introduces a download manager so that file downloads can be recorded and scripted more easily without recourse to sending keystrokes to IE’s standard file download box.

We’ve improved the existing commands so that you can script more tags and use more attributes and added new functions for retrieving data, retrieving the screen position of elements and interfacing with the new native IE functions in Macro Scheduler 13.1.

Oh, and you no longer need to use the mouse to record clicks and form fills!

  • Cleaner, leaner, faster code.
  • More reliable method for waiting for documents to complete loading
  • Download manager for scripting file downloads
  • Better error trapping
  • Ability to set timeout for Clicks and Form Fills etc
  • Click and Form fill functions will wait until target tags exist (within timeout)
  • New functions for extracting data
  • New function to get X,Y screen position of elements
  • No need to use the mouse to identify recorded elements
  • New, improved UI look and feel
  • Runtime component integrates better with Macro Scheduler’s script controller (faster termination, less liable to hang ups when aborting scripts etc)
  • Interface easily with Macro Scheduler’s IEGetTags and IETagEvent function
  • New script function for waiting for text in the page before continuing
  • Functions to automatically set extracted tag buffer size and avoid buffer overruns

More information on WebRecorder here.  WebRecorder 3.0 scripts require Macro Scheduler 13.1 to run.

Enterprise customers with in-date maintenance can already download WebRecorder 3.0 from the registered download area.

Other customers can view upgrade options in their registered download account, or download a trial version here.

Trial Downloads | Registered Downloads | Upgrades

May 20, 2011

Scraping Data From Web Pages

Filed under: Automation,Scripting,Web/Tech — Marcus Tettmar @ 1:02 pm

I’ve seen quite a lot of requests lately from people wanting to know how to extract text from web pages.

Macro Scheduler’s optional WebRecorder add-on simplifies the automation of web pages and includes functions for extracting tables, text or HTML from web page elements. WebRecorder’s Tag Extraction wizard makes it easy to create the code.

Sometimes you can choose a specific HTML element and identify it uniquely via it’s ID or NAME attribute. But other times you might want all the text from the whole page, or you may need to extract the entire page and then parse out the bits you’re interested in using RegEx or some other string manipulation functions.

To extract an entire page I specify the BODY element. If you want to extract data from web pages it does help if you know a little about HTML. And if you do you’ll know that each page has just one BODY element which contains the code making up the visible portion of the page.

Here’s code produced using WebRecorder when navigating to mjtnet.com and using the Tag Extraction wizard to extract the BODY text:

IE_Create>0,IE[0]

IE_Navigate>%IE[0]%,http://www.mjtnet.com/,r
IE_Wait>%IE[0]%,r
Wait>delay

//Modify buffer size if required (you may get a crash if buffer size too small for data) ...
Let>BODY0_SIZE=9999
IE_ExtractTag>%IE[0]%,,BODY,0,0,BODY0,r
MidStr>r_6,1,r,BODY0

MessageModal>BODY0

The macro simply displays just the text in a message box but could be set to pull out the full HTML. You could then parse it with RegEx to get the information you are interested in.

You will need WebRecorder installed for the above to work.

If you don’t have WebRecorder you can do the same with a bit more work using VBScript. Some library functions for doing this can be found here and here.

So here’s the equivalent in VBScript:

VBSTART
Dim IE

'Creates IE instance
Sub CreateIE
  Set IE = CreateObject("InternetExplorer.Application")
  IE.Visible=1
End Sub

'Navigate to an IE instance
Sub Navigate(URL)
  IE.Navigate URL
  do while IE.Busy
  loop
End Sub

'This function extracts text from a specific tag by name and index
'e.g. TABLE,0 (1st Table element) or P,1 (2nd Paragraph element)
'set all to 1 to extract all HTML, 0 for only inside text without HTML
Function ExtractTag(TagName,Num,all)
  dim t
  set t = IE.document.getElementsbyTagname(Tagname)
  if all=1 then
    ExtractTag = t.Item(Num).outerHTML
  else
    ExtractTag = t.Item(Num).innerText
  end if
End Function
VBEND

VBRun>CreateIE
VBRun>Navigate,www.mjtnet.com

VBEval>ExtractTag("BODY",0,0),BodyText
MessageModal>BodyText

But what if you already have a macro which already opens IE, or works against an already open instance of IE? The above macros need to create the IE instance before they can access them and extract data from them. You may have a macro that already starts IE some other way – maybe just by using a RunProgram or ExecuteFile call, or indirectly via some other application. Many times people tackle the extraction of data from such an IE window by sending keystrokes to do a Select-All, Edit/Copy and then use GetClipboard; or even File/Save As to save the HTML to a file. This of course adds time and can be unreliable. So how else can we do it?

Well, this tip shows us a function we can use to attach to an existing IE instance. So let’s use that and then use our ExtractTag function to pull out the BODY HTML:

VBSTART
Dim IE

' Attaches to an already running IE instance with given URL
Sub GetIE(URL)
  Dim objInstances, objIE
  Set objInstances = CreateObject("Shell.Application").windows
  If objInstances.Count > 0 Then '/// make sure we have instances open.
    For Each objIE In objInstances
      If InStr(objIE.LocationURL,URL) > 0 then
        Set IE = objIE
      End if
    Next
  End if
End Sub

'This function extracts text from a specific tag by name and index
'e.g. TABLE,0 (1st Table element) or P,1 (2nd Paragraph element)
'set all to 1 to extract all HTML, 0 for only inside text without HTML
Function ExtractTag(TagName,Num,all)
  dim t
  set t = IE.document.getElementsbyTagname(Tagname)
  if all=1 then
    ExtractTag = t.Item(Num).outerHTML
  else
    ExtractTag = t.Item(Num).innerText
  end if
End Function
VBEND

VBRun>GetIE,www.mjtnet.com

VBEval>ExtractTag("BODY",0,1),BodyHTML
MessageModal>BodyHTML

This snippet assumes a copy of IE is already open and pointing to www.mjtnet.com. The GetIE call creates a link to that IE window and then we use the ExtractTag function to pull out the HTML of the BODY element.

These examples use the BODY element, which will contain everything displayed on the page. As I mentioned before you can be more specific and specify some other element, and with WebRecorder, or a modified version of the ExtractTag VBScript function use other attributes to identify the element (the existing VBScript ExtractTag function shown above just uses the numeric index). WebRecorder tries to make it simple by giving you a point and click wizard, making some assumptions for you, so that you need not fully understand the HTML of the page. But it still helps you understand HTML. Looking at the source of the page you should be able to identify the element you need to extract from. And whether you extract directly from that or extract the BODY and then use RegEx being prepared to delve into the HTML source is going to get you further.

UPDATE: 19th January 2012

As of version 13.0.06 Macro Scheduler now includes a function called IEGetTags. For a given tag type and IE tab this will retrieve an array of tag contents. It can extract just the text, or html of the tags. This example extracts the inner HTML of all DIV elements in the open IE document currently at www.mjtnet.com:

IEGetTags>mjtnet.com,DIV,H,divArr

You can then cycle through each one with a Repeat Until

If>divArr_count>0
  Let>k=0
  Repeat>k
    Let>k=k+1
    Let>this_div_html=divArr_%k%
    .. 
    .. do something with it
    .. e.g. use RegEx or substring searching to determine 
    .. if this is the DIV you want and extract from it
    .. 
  Until>k=divArr_count
Endif

To further identify the tag you are interested in, or find the data you want, you can use RegEx, EasyPatterns, or string functions.

Macro Scheduler 13.0.06 and above also has a function called IETagEvent which will let you simulate a Click on a given tag, focus it, or modify its value. So once you have identified a tag using IEGetTags and your Repeat/Until loop you can click on it, focus it or modify its value (e.g. for form fields).

May 25, 2010

Tweetlib: A DLL Plugin for Tweeting Status Updates via oAuth

Filed under: Announcements,Automation,Scripting,Web/Tech — Marcus Tettmar @ 4:23 pm

As noted yesterday I have been waiting on Twitter to provide xAuth access. They declined, saying it was not appropriate. I’m not really sure why.

No matter, I decided to make a small DLL to simplify Tweeting from Macro Scheduler. It uses the full oAuth interface.

Implementing oAuth in Macro Scheduler code would require lots of VBScript code and would be very complicated (although doable in theory). So instead I decided to create a DLL which you can use in Macro Scheduler to tweet in one line of code.

You can download it here.

And then to post a status update all you need to do is something like this:

Let>message=Hello from Macro Scheduler
LibFunc>%SCRIPT_DIR%\tweetlib.dll,UpdateStatus,r,message,buff,1024

Note that the first time you call UpdateStatus you will be asked to log into Twitter and click “Allow” to authorise Macro Scheduler to access your account. You will then be given a PIN to enter. You only need to do this once. If you ever need to revoke access and start over call the RemoveCredentials function. Your Twitter username and password are NOT stored anywhere. This uses the oAuth authorisation scheme which provides an access token. It is the access token which is stored and this only allows Macro Scheduler to access the API for your account.

The return buffer will contain the XML of the status update operation if successful or an error message if not.

See readme.txt and sample .scp in the zip file. Enjoy.

May 24, 2010

Tweeting from Macro Scheduler Without the API

Filed under: Automation,Scripting,Web/Tech — Marcus Tettmar @ 1:51 pm

A while back I posted an article showing how to Tweet via Twitter‘s API. It uses basic authentication which Twitter plan to turn off in the near future. The alternative, oAuth is awkward for desktop based apps, but xAuth is now available and should be doable in Macro Scheduler. I have requested xAuth access from Twitter and, assuming it’s doable, will try and provide an example once I’ve received it and tried it out.

In the mean time it occurred to me that we don’t really need an API if all we want to do is send a status update. We can do that easily using Macro Scheduler and WebRecorder functions by controlling an instance of Internet Explorer, which can be done in the background.

Below is a script which demonstrates this. It offers a function called LoginToTwitter which need only be called once per session, and an UpdateStatus function to update your status. Just set your Twitter username and password in the first two lines and you should be all set.

Let>TW_USERNAME=XXXXX
Let>TW_PASSWORD=XXXXX

//only need do this once per session
GoSub>LoginToTwitter

GoSub>UpdateStatus,This is a test

GoSub>UpdateStatus,This is a test 2

GoSub>LogOut

// END

//*** SUBROUTINES ***//
SRT>LoadWR
  //load the WebRecorder runtime
  LibLoad>IEAuto.dll,hIE
  If>hIE=0
    MessageModal>Could not load IEAuto.dll, make sure it is in the path or edit the LibLoad line.
    Exit>0
  EndIf
END>LoadWR

SRT>LoginToTwitter
  GoSub>LoadWR
  //open IE
  LibFunc>hIE,CreateIE,ieTwitter,0
  LibFunc>hIE,ShowIE,res,ieTwitter,1

  //log in to Twitter
  LibFunc>hIE,Navigate,r,ieTwitter,http://twitter.com/login
  LibFunc>hIE,WaitIE,r,ieTwitter
  LibFunc>hIE,FormFill,r,ieTwitter,,,session[username_or_email],TW_USERNAME,0
  LibFunc>hIE,FormFill,r,ieTwitter,,,session[password],TW_PASSWORD,submit

  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
END>LoginToTwitter

SRT>UpdateStatus
  LibFunc>hIE,FormFill,r,ieTwitter,,,status,UpdateStatus_VAR_1,submit
  LibFunc>hIE,WaitIE,r,ieTwitter
  Wait>1
END>UpdateStatus

SRT>LogOut
  LibFunc>hIE,KillIE,r,ieTwitter
END>LogOut

You need the IEAuto.DLL library which is installed with WebRecorder.

For a bit of fun the following code copies the currently highlighted text to the clipboard and tweets it. So assigned to a hot key it can be used to tweet any text from any application.

Press CTRL
Send>c
Release CTRL
WaitClipBoard
GetClipBoard>theText
GoSub>UpdateStatus,theText

The real challenge is finding something useful to do with it! 🙂

Older Posts »