Monitoring Website For Keyword With Alert

CyberCitizen · Post by **CyberCitizen** » Mon May 21, 2012 8:52 am

Hi Guys,

I have been asked to look at creating a script that monitors a page (every 5 secs) http://paging1.sacfs.org/live/ for a few keywords and then if the word appears pops up a message box and plays a sound / file.

I have been using MS for a while, however never played much with WebPages.

Also note that the key words will appear on that page for a while, I don't want the alert to keep going off unless the word appears again.

Eg on that page if the word Aldinga appears, display message box, play sound. Waits for new reference of Aldinga to appear.

But I am after more than one word.

Looking at like 3-5 keywords.

Can anyone help.

I have MS v11 & v12. Currently do not have 13 yet. Can't afford it at the moment, I still use v11 for dialogs as I haven't got my head around them in v12.

jpuziano · Post by **jpuziano** » Mon May 21, 2012 10:33 pm

Hi CyberCitizen,

The first thing I would do would be to use HTTPRequest> to pull the source of that web page into variable:

Code: Select all

HTTPRequest>http://paging1.sacfs.org/live/,,GET,,HTMLResponse
MDL>HTMLResponse

The above should work but it doesn't, I get the following error instead:

403 HTTP/1.1 403 Forbidden

However that url works just fine if you hit it from a browser...

Next steps anyone?

jpuziano · Post by **jpuziano** » Mon May 21, 2012 11:05 pm

A quick search of he forums found this post which has a solution supplied by Marcus:
http://www.mjtnet.com/forum/httprequest ... t6649.html

mtettmar wrote:I have also noted the issue with the user agent string. Web sites shouldn't really filter based on user agent, but never-the-less I'll see if we can change the user agent and/or make it over-ridable in a future release.

Yes Marcus, I trust this issue is on the Wish List, but untill then, we have your VBScript work-around... used below.

CyberCitizen, the following will work to pull in the html source for that site:

Code: Select all

VBSTART
Function HTTPGet(URL)
  Set IE = CreateObject("InternetExplorer.Application")
  IE.visible = 0
  IE.navigate URL
  do while IE.Busy
  loop
  HTTPGet = IE.document.documentelement.outerhtml
  IE.quit
  Set IE = Nothing
End Function
VBEND

VBEval>HTTPGet("http://paging1.sacfs.org/live/"),HTMLResult

MessageModal>HTMLResult

Label>check_1
Let>pattern=Aldinga
RegEx>pattern,HTMLResult,0,matches,num,0
If>num=0,check_2
MDL>%pattern% occured %num% time/s in the web page!

Label>check_2
Let>pattern=Adelaide
RegEx>pattern,HTMLResult,0,matches,num,0
If>num=0,check_3
MDL>%pattern% occured %num% time/s in the web page!

Label>check_3
Let>pattern=TRUCK ROLL
RegEx>pattern,HTMLResult,0,matches,num,0
MDL>%pattern% occured %num% time/s in the web page!

I also added three simple checks for various words... this should get you started.

Good luck and take care!

Bob Hansen · Post by **Bob Hansen** » Tue May 22, 2012 3:02 am

How will you check to make sure its not a second copy of the first word that may have been removed? Need some typt of time monitor.

Suppose there are now multiple copies of the word, do you need to monitor that?

What if the word is in a different position because other words have been added or deleted?

You need to provide more specific specs about your constraints. Be exact about what you are looking for, providing examples with time constraints.

CyberCitizen · Post by **CyberCitizen** » Tue May 22, 2012 3:14 am

Thanks jpuziano,

What version of MS are you using? Wondering Marcus ever updated the user agent as it was back in Jan 2011.

I looked at the script you gave me, do I loop the VB part for IE? When I was running it, I was ending up with multiple iexplore processes.

CyberCitizen · Post by **CyberCitizen** » Tue May 22, 2012 3:22 am

Hi Bob,

This page displays all the emergency response messages close to real time. When we get paged to respond to an emergency you will see our response message appear on this list.

The message will appear here approx 5-10 seconds after we are paged, however what we find is that the ambulance is paged before us, I am looking at a way of monitoring this page for the suburbs eg Aldinga.

So if the ambulance gets paged to Main South Road Aldinga we are most likely looking at getting called to the same job as that is a high impact area.

We are basically trying to get a ready response system so if we get an alert we can get our keys, car & gear ready so if we get a page we are out the door 30 seconds faster. Believe me it makes a big difference.

But yes as you mentioned I don't want the system to keep sending alerts if the word appears again and again.

Normally the same response message will go out 2-3 times, so 3x alerts are fine, however as it gets moved down the page with newer messages its position would change & we wouldn't want it alerting again until something different appeared.

I might be clutching at straws trying to get something like this going, but we are a volunteer brigade and anything we can do to assist our local residents we will try.

Bob Hansen · Post by **Bob Hansen** » Tue May 22, 2012 3:53 pm

Here is an overview approach that may help ...

Every five seconds Use HTTP to get the file.
USe Regex to find the line with the first instance of each key word
Parse out that line to grab the date/time
write the keyword and date/time to an ASC file to use as a database.
Run a SQL to count how mucn time elapsed for the last four instances of each keyword. If elapsed time is less than some defined period, then sound alarm.
========================

If you are creating the log, you could also take a similar approach when the log file is created. When creating the log, do the calculation for elapsed time then, and add a special char string to the line. Now you only need to search for that string in the last five seconds. Would work for all keywords.

jpuziano · Post by **jpuziano** » Tue May 22, 2012 4:20 pm

CyberCitizen wrote:Thanks jpuziano,

What version of MS are you using? Wondering Marcus ever updated the user agent as it was back in Jan 2011.

The script worked for me under MS 11 and 13 (that was what I have available at home) so I'm thinking it should work fine in MS 12 as well.

No, Marcus must not have updated the user agent yet because the VBScript workaround is still required in MS version 13.1.06 which is currently the latest version available.

I believe this issue is on the Wish List... Marcus can you confirm this?

CyberCitizen wrote:I looked at the script you gave me, do I loop the VB part for IE? When I was running it, I was ending up with multiple iexplore processes.

You should never loop the lines between:

VBSTART
and...
VBEND

Those lines are best placed right at the top of your script as they should only ever run once and you would never loop back to run them again... its just not needed and may actually cause problems.

The following line, you would call as many times as you need to... each time you want to fetch the html source as it exists *right now*:

VBEval>HTTPGet("http://paging1.sacfs.org/live/"),HTMLResult

The following lines in that VBScript function should be closing the IE instance it created just before the function ends...

IE.quit
Set IE = Nothing

Grabbing the html source is the easy part... Keeping track of instances of your keywords and when they occurred is going to be more tricky but you already have some good suggestions from Bob on that.

Keep in touch and let us know how you do on this one... it sounds like a very worthwhile project.

jpuziano · Post by **jpuziano** » Tue May 22, 2012 4:57 pm

CyberCitizen wrote:Thanks jpuziano,

What version of MS are you using? Wondering Marcus ever updated the user agent as it was back in Jan 2011.

The script worked for me under MS 11 and 13 (that was what I had at home) so I'm thinking it should work fine in MS 12 as well.

No, Marcus must not have updated the user agent yet because the VBScript workaround is still required in MS version 13.1.06 which is currently the latest version available.

I believe this issue is on the Wish List... Marcus can you confirm this?

CyberCitizen wrote:I looked at the script you gave me, do I loop the VB part for IE? When I was running it, I was ending up with multiple iexplore processes.

You should never loop the lines between:

VBSTART
and...
VBEND

Those lines are best placed right at the top of your script as they should only ever run once and you would never loop back to run them again... its just not needed and may actually cause problems.

The following line you'd call as many times as you need to... each time you want to fetch the html source as it exists *right now*:

VBEval>HTTPGet("http://paging1.sacfs.org/live/"),HTMLResult

The following lines in that VBScript function should be closing the IE instance it created just before the function ends...

IE.quit
Set IE = Nothing

Grabbing the html source is the easy part... Keeping track of instances of your keywords and when they occurred is going to be more tricky but you already have some good suggestions from Bob on that.

Keep in touch and let us know how you do on this one... it sounds like a very worthwhile project.

CyberCitizen · Post by **CyberCitizen** » Mon Jun 25, 2012 7:02 am

Thanks for the help guys.

I am getting further along. I am writing the results to a file every 5 seconds.

Then I am reading a line number to check if it contains the keywords specified. If it exists, it plays an alert & reads the message.

The issue I have now is I am trying to find what line number the HTML tag is on so I can reference it with the rest of the script.

What I am looking for is the first instance of this in the document

Code: Select all

.TD class=message.

Replace the . with either forum wouldn't let me post it otherwise.
That is the line number I need to capture. I think I have a way of doing it, by reading each like and then checking with RegEx if that string existed if so that is my line number to start at.

Thoughts?

Code: Select all

Let>APP_TITLE=CFS Response Monitor
Let>MSG_HEIGHT=320
Let>MSG_WIDTH=500

VBSTART
Function HTTPGet(URL)
  Set IE = CreateObject("InternetExplorer.Application")
  IE.visible = 0
  IE.navigate URL
  do while IE.Busy
  loop
  HTTPGet = IE.document.documentelement.outerhtml
  IE.quit
  Set IE = Nothing
End Function
VBEND

VBSTART
Sub Speak(text)
 Dim Talk
 Set Talk = CreateObject("SAPI.SpVoice")
 Talk.Speak text
End Sub
VBEND

VBEval>HTTPGet("http://paging1.sacfs.org/live/"),HTMLResult

DeleteFile>%TEMP_DIR%CFSPagingData.txt
WriteLn>%TEMP_DIR%CFSPagingData.txt,WriteResult,%HTMLResult%
//It is here that I need to capture the line number in that case its 20
ReadLn>%TEMP_DIR%CFSPagingData.txt,20,FirstMSG

CyberCitizen · Post by **CyberCitizen** » Tue Jun 26, 2012 12:58 am

Ok I worked it out, got it working mostly to what I am looking for, the issue I have now is I got the page suspended as I guess there were to many hits.

Maybe I will just have to monitor it for changes on screen then capture the text & then check it via regex.