Best practice for pulling data from central source

General Macro Scheduler discussion

Moderators: Dorian (MJT support), JRL

Post Reply
Warren
Pro Scripter
Posts: 83
Joined: Sun Oct 08, 2017 11:57 pm

Best practice for pulling data from central source

Post by Warren » Mon Oct 09, 2017 5:10 am

I'm looking at setting up some vps with macros that do things like read and write to .csv's.

I've got that much working, but it may be a better idea to have them all pull from the same source so I can update and check files in one place.

Basically, I'd be just hosting relatively small files. What's the most reliable way to do this that returns the fewest connection errors? Is there a quick and easy way to just host some .csv files on a cdn somewhere and use the site's address as part of the filepath? Do I actually need a website, or is there a simpler way to just store and access the files from a cloud server or similar?

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: Best practice for pulling data from central source

Post by Marcus Tettmar » Mon Oct 09, 2017 2:07 pm

Not entirely sure I know what you are asking. You can pull data in a number of ways including from a SQL data source using DBOpen/DBQuery etc. As for reading from files, where you put those files is up to you.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

Warren
Pro Scripter
Posts: 83
Joined: Sun Oct 08, 2017 11:57 pm

Re: Best practice for pulling data from central source

Post by Warren » Mon Oct 09, 2017 5:56 pm

Just pleading ignorance about the best way to host the files. How are other people doing this? Can I just dump them on some cloud service, or do I need to establish a site so there's a defined address/filepath? Seems like there might be some sort of service dedicated to this, but I'm not sure what terminology to look up.

How are others here hosting their files if they have multiple machines that need to access a single source reliably?

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: Best practice for pulling data from central source

Post by Marcus Tettmar » Tue Oct 10, 2017 10:27 am

I guess most people will be using network shares. Not really sure what files we're talking about or how directly relevant to Macro Scheduler this is.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

Warren
Pro Scripter
Posts: 83
Joined: Sun Oct 08, 2017 11:57 pm

Re: Best practice for pulling data from central source

Post by Warren » Sat Oct 28, 2017 9:18 pm

It's possible I'm not asking it correctly, but it's becoming more and more obvious to me that I'm doing a few things backward because I simply haven't seen the right way to do them yet, and that working out some of these details are very important to properly integrating Macro Scheduler. It's also possible that when I dig into remote controller, I'll have better solutions anyway, but for now, trying to understand issues as I see them in front of me.

I'll be specific. I want each remote machine upon startup to automatically launch a script that is centrally stored so I can change the script once, and each machine will automatically run the new version next time it comes up.

I've never dealt with anything other than local filepaths, and I've never hosted anything on any kind of shared or cloud drive (other than dropbox), so I'm just trying to connect the dots and learn the right terminology.

I would think I need to do something like have a script in the startup folder of each machine (so it launches on startup) which just launches a separate script via an external https:// filepath which refers to a central external drive or cloud storage location. I'm assuming that the external script would then load into local ram allowing it to be run locally even though the file is hosted externally. (image_dir issues aside)

Do I at least have that process right? If so, what's the most reliable way to:

1) Store the main (external) script so each machine can access it quickly and ultra-reliably. (shared drive, cloud hosting, network share, etc)

2) Refer to the external script via the local launcher script so it loads into local ram and executes. (command type, filepath)

3) Launch the local launcher script on Windows startup. (EXE in startup folder as described, or some other method?)

Hope that's clearer.

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: Best practice for pulling data from central source

Post by Marcus Tettmar » Sun Oct 29, 2017 11:01 am

Yes, just put your script on a network share and stick a shortcut in each PC's startup folder. Or if you're using a Domain you could use a login script in the profile or group policy.

You might want to talk to your system administrator because this is nothing specific to Macro Scheduler - it's standard stuff - people do it all the time, regardless of what technology is running the actual script.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

Warren
Pro Scripter
Posts: 83
Joined: Sun Oct 08, 2017 11:57 pm

Re: Best practice for pulling data from central source

Post by Warren » Wed Nov 01, 2017 4:19 am

No network admin. No network really... just 100 rented virtual machines spread across multiple vendors and multiple continents. The only common thread is the single RDP client on my home computer that I use to access them.

Rolling out a networking setup that fits the MS workflow seems to me to be a critical part of using MS to do anything on a larger scale. I tried setting up a shared folder on my home machine, and connecting it to each machine via their network settings. It was so slow that it was nowhere near usable. I need the opposite of that... something that will never have down time, and always be accessed as quickly as possible.

If I launch a centrally stored master script remotely from each machine as described earlier, is the entire script loaded into the local ram? If so, are the images all loaded into ram as well, or are they called via their filepath as the are referenced individually? If it's the latter, I would guess that the script would break if there was a hiccup in the connection. That's why I was thinking I'd need some kind of super fast and reliable cdn or redundant cloud solution.

Am I misunderstanding that? Is the full script along with all images just loaded into ram upon launch so it doesn't matter how anything is networked once it loads?

If not, does that mean that I at least need to have the images stored one each local machine to avoid scripts breaking due to connection issues?

Or am I missing that the remote controller (which I haven't had a chance to look at yet) solves all these issues and more?

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: Best practice for pulling data from central source

Post by Marcus Tettmar » Wed Nov 01, 2017 7:39 am

Dropbox? OneDrive?

Scripts and images are files - nothing is in RAM until used.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

Warren
Pro Scripter
Posts: 83
Joined: Sun Oct 08, 2017 11:57 pm

Re: Best practice for pulling data from central source

Post by Warren » Thu Nov 02, 2017 12:09 am

Oh, I think I get it now. I had looked at dropbox a while back, but missed one critical feature. I was looking at it like cloud storage and hung up on the remote connection thing... like what happens if I'm trying to access a remote file, and there's a hiccup in connection.

Digging into it, it looks like I missed that it's possible to sync the files so they can actually be stored locally on each machine avoiding connection issues, but also have that folder auto-update when I swap out a script. If I understand correctly now, that should be exactly what I need, so I guess I just need to put a script in each startup folder that says to launch a script by a given name in the dropbox folder, then swap out the script by that name on my compiling machine, and each local folder should update... I think.

If that's correct, and nothing is loaded into ram, so each machine is constantly going back to the script and image dir in that dropbox folder while the script is running, then II guess I'd just have to figure out how to deal with the fact that at any given moment, the script I want to swap out will currently be running on a number of machines.

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: Best practice for pulling data from central source

Post by Marcus Tettmar » Thu Nov 02, 2017 10:32 am

Maybe have your master script Include> the script in the dropbox folder, allowing you to update that script whenever you want. Include will pull it into memory when it runs.

Go a step further - How about put your scripts in a dropbox folder, but don't have Macro Scheduler directly linked there. Instead have your "master" script check to see if an update exists in that dropbox folder (look at the file date, or use some naming convention?) and if it finds a new one it will COPY that file to a local file and THEN use Include> or Macro> to run it.

Or use an ODBC database in the cloud (e.g. MySQL - it's free). Put the code there, have it store the version number. Master script uses SQL to see if there's an update, if so pulls the code down and saves to a file, and runs that. Or use FTP to retrieve the latest code. Master script first connects via FTP, retrieves the latest script file, then runs/includes it.

So many ways you could do this. Don't overthink it ;-)
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

Warren
Pro Scripter
Posts: 83
Joined: Sun Oct 08, 2017 11:57 pm

Re: Best practice for pulling data from central source

Post by Warren » Thu Nov 02, 2017 3:53 pm

I'll play with it. I think the first option may be enough, though. So long as it's fully loaded into ram, it shouldn't matter if I switch it out while running (except switching image folder). It will just run the old script until it shuts itself down, and on reboot, the launcher script will load automatically again, but this time include will fetch the new script. Only issue I see there is that include will break if it's trying to fetch during the brief time it takes me to do the actual script swap. Just telling it to wait or reboot again if the include breaks should fix that, though.

Post Reply
Sign up to our newsletter for free automation tips, tricks & discounts