Screen scraping values from a page

Technical support and scripting issues

Moderators: Dorian (MJT support), JRL

Post Reply
mightycpa
Automation Wizard
Posts: 343
Joined: Mon Jan 12, 2004 4:07 pm
Location: Vienna, VA

Screen scraping values from a page

Post by mightycpa » Fri Feb 10, 2023 5:06 pm

Hi all,

I'm trying to grab some values off a public web page, not sure how to approach this. I looked at the underlying HTML, and while the container is pretty well defined, there don't seem to be any values I can capture.

The web page: https://finance.yahoo.com/chart/SPY#eyJ ... QifX19fQ--

Here's the things I want to capture:
Image

Specifically, at the top left, I want to capture the current price. Then on the right, you can see colored boxes that have prices in them. They move up and down sometimes, and sometimes one will overlay another, so they might be a little difficult to grab. The long url ensures that the rectangle that I want to search in for those values will always remain the same size and in the same position in the same browser, and the colors will not vary either. Only the numbers.

I want to capture all the values in all the colors into variables: Red, Yellow, Fuschia, Kelly Green, Orange, Lime, Dirty Yellow, Deep Green.

Image

Can somebody tell me how? I've never been too good at capturing this web stuff. I'm browser agnostic if that helps. Thanks in advance.
"A facility for quotation covers the absence of original thought." - Lord Peter Wimsey

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: Screen scraping values from a page

Post by Grovkillen » Fri Feb 10, 2023 5:37 pm

Let me guess, they use canvas elements to draw these graphics? If so it's possible to inject Javascript code to snatch the info instead of using image recognition.
Let>ME=%Script%

Running: 15.0.24
version history

mightycpa
Automation Wizard
Posts: 343
Joined: Mon Jan 12, 2004 4:07 pm
Location: Vienna, VA

Re: Screen scraping values from a page

Post by mightycpa » Fri Feb 10, 2023 5:59 pm

Thanks. It's difficult for me to say, because I don't understand what you're talking about!! :D

That said, when I look at the HTML, I think these two things are what make this page work:
1. The red box highlights for a few seconds when I hover over any of the colored prices
2. I think the ratechange object is the listener for the changing price.

Image

It doesn't help me, but does is help you?
"A facility for quotation covers the absence of original thought." - Lord Peter Wimsey

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: Screen scraping values from a page

Post by Grovkillen » Fri Feb 10, 2023 8:00 pm

Looks doable. But yes, you would need to inject Javascript and perhaps output the values to a external file or a custom HTML element.
Let>ME=%Script%

Running: 15.0.24
version history

mightycpa
Automation Wizard
Posts: 343
Joined: Mon Jan 12, 2004 4:07 pm
Location: Vienna, VA

Re: Screen scraping values from a page

Post by mightycpa » Mon Feb 13, 2023 4:54 pm

I've asked some people to look into it, one tells me that the values are encrypted. Have a few more inquiries out there.
"A facility for quotation covers the absence of original thought." - Lord Peter Wimsey

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: Screen scraping values from a page

Post by Grovkillen » Mon Feb 13, 2023 5:55 pm

Encrypted... How so? They are clearly visible and I don't see how they would be able to hide them except if they write them as bits/pixel values. I doubt it.
Let>ME=%Script%

Running: 15.0.24
version history

hagchr
Automation Wizard
Posts: 327
Joined: Mon Jul 05, 2010 7:53 am
Location: Stockholm, Sweden

Re: Screen scraping values from a page

Post by hagchr » Mon Feb 13, 2023 8:23 pm

If you click on "Try the new and improved charts" and "join beta" then it seems they have added a "Table View" where you can access all the Bollinger bands per day in table format. This should make your life much easier.

mightycpa
Automation Wizard
Posts: 343
Joined: Mon Jan 12, 2004 4:07 pm
Location: Vienna, VA

Re: Screen scraping values from a page

Post by mightycpa » Fri Feb 24, 2023 4:57 am

I just saw that today and joined up! They sure do make it easy!
"A facility for quotation covers the absence of original thought." - Lord Peter Wimsey

mightycpa
Automation Wizard
Posts: 343
Joined: Mon Jan 12, 2004 4:07 pm
Location: Vienna, VA

Re: Screen scraping values from a page

Post by mightycpa » Fri Feb 24, 2023 5:02 am

@Grovkillen,

I don't know, I was just passing along what they told me. What I did learn is that if you just download the html, you won't find those values in there anywhere. So I suppose they are delivering pictures rather than text data. Maybe those pictures are created server-side, I'm not sure. I do know that none of the values can be captured as text from the screen.

But now, as hagchr points out, they're providing the data in text format in a table, which is a lot easier to deal with.
"A facility for quotation covers the absence of original thought." - Lord Peter Wimsey

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: Screen scraping values from a page

Post by Grovkillen » Fri Feb 24, 2023 1:49 pm

Great you found a way. But for the future, those values are stored within the canvas and if you know a little Javascript it's relativity trivial to get the data.
Let>ME=%Script%

Running: 15.0.24
version history

Post Reply
cron
Sign up to our newsletter for free automation tips, tricks & discounts