ChromeDriver 404 on URL from Variable

Technical support and scripting issues

Moderators: Dorian (MJT support), JRL

Post Reply
NickD
Pro Scripter
Posts: 58
Joined: Fri Sep 23, 2016 2:17 pm

ChromeDriver 404 on URL from Variable

Post by NickD » Thu Apr 09, 2020 3:52 pm

I am working on a script that logs into my work diary, and exports my schedule to a csv file that I can then upload to google calendar to make my schedule available on my android phone.

Each event on my work diary has its own details page that I need to be able to scrape. My script successfully navigates the work diary and extracts the detail page URLs, then writes them to a csv file. This csv file is then exploded to an array, but when I try to navigate to the detail page URLS, a 404 error message is thrown by MsChed.

Example url: https://portals.securitas.uk.com/Employ ... d=46998903

I think this could be a character issue with the URL string, but can't figure out how to fix it. I already tried using double quotes around the variable which contains the URL in ChromeNavigate, but that did not help. Any ideas?

Code: Select all

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Log In
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////


  ChromeStart>session_id
  WindowAction>1,data:*
  ChromeNavigate>session_id,url,https://portals.securitas.uk.com/Employee/Account/Login?ReturnUrl=%2FEmployee%2F


  Let>EmailSelector={"#Email"}
  ChromeFindElements>session_id,css selector,EmailSelector,EmailField
  ChromeSetElementValue>session_id,EmailField_1,%Email%


  Let>PasswordSelector={"#Password"}
  ChromeFindElements>session_id,css selector,PasswordSelector,PasswordField
  ChromeSetElementValue>session_id,PasswordField_1,%Password%

  Let>LoginBtnSelector={"#btnLogin"}
  ChromeFindElements>session_id,css selector,LoginBtnSelector,LoginBtn
  ChromeElementAction>session_id,LoginBtn_1,click

  Label>WaitLogin
  Let>DiarySelector={"body > div.container > main > div.PortalCenterContainer > div:nth-child(2)"}
  ChromeFindElements>session_id,css selector,DiarySelector,DiaryBtn

  If>DiaryBtn_count<1
  Wait>1
  Goto>WaitLogin
  Endif


////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Navigate to Diary
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////


  Let>DiarySelector={"body > div.container > main > div.PortalCenterContainer > div:nth-child(2)"}
  ChromeFindElements>session_id,css selector,DiarySelector,DiaryBtn
  ChromeElementAction>session_id,DiaryBtn_1,click

  Label>WaitDiaryLoad
  Let>NextWeekArrowSelector={"#DiaryDateForm > div > div > span.icon-arrow-right2"}
  ChromeFindElements>session_id,css selector,NextWeekArrowSelector,NextWeekArrow
  If>NextWeekArrow_count<1
  Wait>1
  Goto>WaitDiaryLoad
  Endif
  


////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Delete Temporary Files
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////


  IfFileExists>%BMP_DIR%\event-urls.csv
  DeleteFile>%BMP_DIR%\event-urls.csv
  Endif


////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Extract Event URLs for the Next 4 Weeks to Array
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////


  Let>week=1
  Label>ExtractWeekEvents


  ChromeGetInfo>session_id,source,theSource


  // Extract Events
  Let>pattern=(?<=DutyConfirmed" onclick="location.href=').+?(?=')
  RegEx>pattern,%theSource%,0,event_array,event_matches,0


  Let>k=0
  While>k<event_matches
  
    Let>k=k+1
    Let>url=https://portals.securitas.uk.com
    Let>eventurl=event_array_%k%
    Concat>url,eventurl
    
    WriteLn>%BMP_DIR%\event-urls.csv,WrLnRes,%url%

  EndWhile


  If>week<4
  
    Let>NextWeekArrowSelector={"#DiaryDateForm > div > div > span.icon-arrow-right2"}
    ChromeFindElements>session_id,css selector,NextWeekArrowSelector,NextWeekArrow
    ChromeElementAction>session_id,NextWeekArrow_1,click
    
    Let>week=week+1
    Goto>ExtractWeekEvents
  
  Endif
  

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Explode Event URL csv file to Array
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////


  CSVFileToArray>%BMP_DIR%\event-urls.csv,url_array



////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Extract Event Details & Write to CSV
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////


  Let>k=1
  While>k<url_array_count
  
  Let>jobdetails=url_array_%k%_0


  //Throws 404 Error Despite Valid URL
  ChromeNavigate>session_id,url,"%jobdetails%"

  
  Let>k=k+1
  
  EndWhile


/*
(Google Galendar CSV Headers, First 2 Required, Rest Optional)

Subject
The name of the event, required.
Example: Final exam

Start Date
The first day of the event, required.
Example: 05/30/2020

Start Time
The time the event begins.
Example: 10:00 AM

End Date
The last day of the event.
Example: 05/30/2020

End Time
The time the event ends.
Example: 1:00 PM

All Day Event
Whether the event is an all-day event. Enter True if it's an all-day event, and False if it isn't.
Example: False

Description
Description or notes about the event.
Example: 50 multiple choice questions and two essay questions 

Location
The location for the event.
Example: "Columbia, Schermerhorn 614"

Private
Whether the event should be marked private. Enter True if the event is private, and False if it isn't.
Example: True

*/

User avatar
Grovkillen
Automation Wizard
Posts: 1009
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: ChromeDriver 404 on URL from Variable

Post by Grovkillen » Thu Apr 09, 2020 4:04 pm

Those % in the URL will be taken for a variable I would think.
Let>ME=%Script%

Running: 15.0.24
version history

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: ChromeDriver 404 on URL from Variable

Post by Marcus Tettmar » Fri Apr 10, 2020 8:13 am

This doesn't look right:

ChromeNavigate>session_id,url,"%jobdetails%"

You have quotes. Do you mean:

ChromeNavigate>session_id,url,%jobdetails%

In fact you shouldn't need the % symbols here so just this should do:

ChromeNavigate>session_id,url,jobdetails
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

NickD
Pro Scripter
Posts: 58
Joined: Fri Sep 23, 2016 2:17 pm

Re: ChromeDriver 404 on URL from Variable

Post by NickD » Fri Apr 10, 2020 1:44 pm

Hi Marcus,

I tried that first, then wrapped in percentage symbols, then with quotes to try and define the variable as a string. Every approach returned a 404 error modal.

When I use:

Code: Select all

    StringReplace>url,=,&#61;,url
    StringReplace>url,?,&quest;,url


To encode the "?" and the "=" chars in the URLs, MSched throws an I/O error 32 instead of a 404 error. If I change the script to use IE it works fine...

https://portals.securitas.uk.com/Employ ... d=46998903

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: ChromeDriver 404 on URL from Variable

Post by Marcus Tettmar » Fri Apr 10, 2020 2:18 pm

Do you get the same issue when hardcoding the URL:

Code: Select all

ChromeNavigate>session_id,url,https://portals.securitas.uk.com/Employee/Diary/DutyDetail?id=46998903
I don't think it's anything to do with the encoding or characters - it follows the same format as your initial URL which I assume is working?

Code: Select all

ChromeNavigate>session_id,url,https://portals.securitas.uk.com/Employee/Account/Login?ReturnUrl=%2FEmployee%2F
You have that in line 3. The URL you are having issues with has nothing different about it.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

Post Reply
Sign up to our newsletter for free automation tips, tricks & discounts