HTTPRequest to Access Subscribed Contents

armsys · Post by **armsys** » Sun May 26, 2013 3:56 am

How to use HTTPRequest to retrieve content from password-protected paid web pages?

CyberCitizen · Post by **CyberCitizen** » Mon May 27, 2013 12:20 am

I believe that its only available that way for basic web auth, anything else you would need to use webrecorder.

For basic authentication where a username and password is required by the server before the request will complete put the username and password in the URL using the following format: http://username:[email protected]/etc.etc

armsys · Post by **armsys** » Mon May 27, 2013 2:53 am

CyberCitizen,
Good morning Australia!
I did try out http://username:[email protected]/etc.etc.
It doesn't return the paid content. I guess the content is protected by some Java tricks.
Thanks for your help.

CyberCitizen · Post by **CyberCitizen** » Mon May 27, 2013 8:46 am

HTTPRequest only really works on sites that give you this type of logon prompts (not web based ones).

armsys · Post by **armsys** » Mon May 27, 2013 8:56 am

CyberCitizen wrote:HTTPRequest only really works on sites that give you this type of logon prompts (not web based ones)

Ok, now I can see what you see.
BTW, it isn't a major issue. MS can simulate keyboard typing. Without HTTPRequest, I still can use mouse+keyboard to extract the contents.
Are you an actual fire fighter as signed?

CyberCitizen · Post by **CyberCitizen** » Tue May 28, 2013 2:31 am

http://www.youtube.com/watch?v=9vJzKbHY7ts

Yes I am, keeps me busy.

Post by **Marcus Tettmar** » Wed May 29, 2013 7:48 am

Not entirely true. HTTPRequest can perform POST operations. So if the website login is a FORM which does a POST, then you can reconstruct that POST using HTTPRequest.

That will get you logged in to the next page. You may then need to "scrape" a session ID out of the page to use with subsequent operations to get data further down the line.

But if the site uses a browser session state or cookie then you won't be able to use HTTPRequest.

In this case it is better to use the WebRecorder functions to control an IE instance which will maintain the session state.

armsys · Post by **armsys** » Wed May 29, 2013 9:24 am

Marcus Tettmar wrote:But if the site uses a browser session state or cookie then you won't be able to use HTTPRequest.

That's exactly the problem I encounter. The websites in question such as Bloomberg provide paid financial information. When I look up their source code (HTML), besides a bunch of Javascript, it doesn't contain any data which are shown my monitor. That's amazing!