HTTPRequest proxy problem

pasdad1 · Post by **pasdad1** » Thu Oct 25, 2007 9:06 pm

I'm having problems with HTTPRequest (using MS 8.0.1) talking correctly to my Squid 2.6 proxy server. This is the sample program I'm running

Code: Select all

 Let>HTTP_TIMEOUT=10
 Let>Q=http://tess2.uspto.gov/bin/gate.exe?p_lang=english&p_d=trmk
 HTTPRequest>Q,,GET,,XMLResponse,192.168.3.3,3128,adroege,482adr
 MessageModal>output=%XMLResponse%

This produces the following output
 output=407 HTTP/1.0 407 Proxy Authentication Required

.

I'm certain the proxy parms are correct, as they are what I'm using in the proxy section of IE when browsing the internet. Seems if HTTPRequest is not sending the authentication in a form Squid is able to read correctly.

I've used this a while back with no problems.... since that time they have changed the proxy to a newer version... but it has not been a problem with any other programs or browsers I'm using.... so it comes back to HTTPRequest.

Post by **Marcus Tettmar** » Thu Oct 25, 2007 9:17 pm

Can you provide the proxy server version and do you know the authentication method it is expecting?

Since it works with IE, you could use this method as a workaround:
http://www.mjtnet.com/forum/viewtopic.php?t=1655

pasdad1 · Post by **pasdad1** » Thu Oct 25, 2007 9:25 pm

The version of the proxy is reported as (squid/2.6.STABLE6)

I don't know what authentication method it is using currently.

pasdad1 · Post by **pasdad1** » Fri Oct 26, 2007 2:11 pm

Update:

I believe the proxy server is using basic authentication. This is usually the only type browsers are capable of. Username and password are sent in a header after being base64 encoded.

Post by **Marcus Tettmar** » Fri Oct 26, 2007 2:14 pm

in which case it is hard to see why Macro Scheduler can't cope.

Did you try the workaround I posted that uses IE?

pasdad1 · Post by **pasdad1** » Tue Jul 22, 2008 2:31 pm

I'm still having the issues with HTTPRequest not talking correctly
to my web proxy - however I've now had the time to document the
problem in more detail.

I installed an open-source java program
http://grinder.sourceforge.net/

Which has as part of it a "debugging proxy" -- to intercept
and log all web communication. In this case I change the proxy
info in Macro Scheduler to point to this debug proxy, and
the debug proxy passes all the info it receives to the "real"
proxy on my network and logs everything.

Here is the log of all the communication from running my
little test macro.

It seems as if HTTPRequest is NOT sending my authentication
properly.

In this case I really need HTTPRequest to work. I don't want
to script IE.

Code: Select all

----------------PROXY DEBUG START-----------------------

--- localhost:3444->www.google.com:80 opened --
--- http://www.google.com:80->localhost:3444 opened --
------ localhost:3444->www.google.com:80 ------
GET / HTTP/1.1
Content-Type: text/html
Host: http://www.google.com
Accept: text/html, */*
User-Agent: Mozilla/3.0 (compatible; Indy Library)


------ http://www.google.com:80->localhost:3444 ------
HTTP/1.0 407 Proxy Authentication Required
Server: squid/2.6.STABLE6
Date: Tue, 22 Jul 2008 13:44:35 GMT
Content-Type: text/html
Content-Length: 1313
Expires: Tue, 22 Jul 2008 13:44:35 GMT
X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
Proxy-Authenticate: Basic realm="Squid proxy-caching web server"
X-Cache: MISS from guardian.adclegal.com
X-Cache-Lookup: NONE from guardian.adclegal.com:3128
Via: 1.0 guardian.adclegal.com:3128 (squid/2.6.STABLE6)
Proxy-Connection: close

<DOCTYPE>
<HTML><HEAD><META>
<TITLE>ERROR: Cache Access Denied</TITLE>
<STYLE><BODY></STYLE>
</HEAD>
<BODY>
<H1>ERROR</H1>
<H2>Cache Access Denied</H2>
<HR>
<P>
While trying to retrieve the URL:
<A>http://www.google.com/</A>
<P>
The following error was encountered:
<UL>
<LI>
<STRONG>
Cache Access Denied.
</STRONG>
</UL>
</P>

<P>Sorry, you are not currently allowed to request:
<PRE>    http://www.google.com/</PRE>
from this cache until you have authenticated yourself.
</P>

<P>
You need to use Netscape version 2.0 or greater, or Microsoft Internet
Explorer 3.0, or an HTTP/1.1 compliant browser for this to work.  Please
contact the <A>cache administrator</a> if you have
difficulties authenticating yourself or 
<A>change</a> your default password.
</P>

<BR>
<HR>
<ADDRESS>
Generated Tue, 22 Jul 2008 13:44:35 GMT by guardian.adclegal.com (squid/2.6.STABLE6)
</ADDRESS>
</BODY></HTML>

--- localhost:3444->www.google.com:80 closed --
--- http://www.google.com:80->localhost:3444 closed --
----------------PROXY DEBUG END-----------------------

Here is the Macro Scheduler Macro I used to test with:

Code: Select all

----------------MACRO START-----------------------
Let>HTTP_TIMEOUT=8
HTTPRequest>http://www.google.com,,GET,,XMLResponse,127.0.0.1,8001,proxyuser,proxypass
MessageModal>output=%XMLResponse%
----------------MACRO END-----------------------

Notes:
1.) I blanked out my "real" proxyuser/proxypass for this posting
2.) 127.0.0.1:8001 is the address/port of my debugging proxy. It passes
all communication to my "real" proxy and logs everything.

Thanks!

Post by **Marcus Tettmar** » Tue Jul 22, 2008 4:33 pm

Are you sure the proxy is only requiring basic authentication? This is the only authentication Macro Scheduler's HTTPRequest function supports. Can you find out what kind of authentication your proxy is requiring?

pasdad1 · Post by **pasdad1** » Tue Jul 22, 2008 4:45 pm

Copied from the response of the proxy in my previous post ....

Code: Select all

Proxy-Authenticate: Basic realm="Squid proxy-caching web server"

Yes, it appears it is looking for basic authentication.

Shouldn't HTTPRequest be sending the username/password base64 encoded in the headers it sends to the proxy? It doesn't show up in the log of the output I captured.

pasdad1 · Post by **pasdad1** » Tue Jul 22, 2008 6:05 pm

Here is more info to support my theory that HTTPRequest is not sending correct proxy authentication...

This is what my debug proxy captured when I made a simple search on google from IE7 browser. Notice the line which reads:
Proxy-Authorization: Basic YWRyb2VnZTo0ODJhZHI=

HTTPRequest doesn't send this when it constructs it's messages.

Code: Select all

GET /search?hl=en&q=test&btnG=Google+Search HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/msword, application/x-shockwave-flash, */*
Referer: http://www.google.com/
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
Proxy-Connection: Keep-Alive
Host: http://www.google.com
Cookie: PREF=ID=b3d2be9fedffc8ff:TM=1202741833:LM=1202741833:S=SVUjdUQG9mZxvRnY; NID=11=p-20NcHjInYABBnOnPAo7SoP29KyQ39t9lfst3iMkoec8uVn2D-JFdeW3imkzPpH3HPAwFK3KZcH8nh_9DO7gP_18f3mxywVaHyWJY68VXuuJLf2s7O7iujqpZHW35ev
Proxy-Authorization: Basic YWRyb2VnZTo0ODJhZHI=

Thanks!

Post by **Marcus Tettmar** » Fri Jul 25, 2008 9:22 am

Thanks for the info and the additional support ticket. I believe we have fixed this issue and I have sent you a download link to test the fix.

pasdad1 · Post by **pasdad1** » Fri Jul 25, 2008 7:21 pm

Thank you Marcus !

This fix worked.

jpuziano · Post by **jpuziano** » Sat Jul 26, 2008 4:59 am

Hi Marcus,

Will there be a maintenance release of Macro Scheduler Pro that has this fix?

Take care

Post by **Marcus Tettmar** » Sat Jul 26, 2008 5:10 am

Of course there will. Just needed confirmation from pasdad1 that his problem was solved first. We'll release this publicly next week.

HTTPRequest proxy problem

HTTPRequest proxy problem

Proxy version

Proxy method

HTTPRequest Proxy Problems

HTTPRequest not working with Squid proxy basic authenticatio

HTTPRequest not sending basic authentication

Thank you Marcus !