Large log files with imbedded images

[Bob Hansen]

I just finished modifying some log files that were enormously large, and the result of some really small scripts.

I reduced one log to the last macro run only. The file size is over 5Mb, and only consists of 42 lines of script. But there are 9 imbedded images beneath the end of the code. I forgot the size originally, but had more than 20 copies of the macro running. Imagine if this was run multiple times per day!

When I looked inside it appears that the entire content of imbedded images are being recorded in the logs. Is there a way to just have the log reference the image data reference name vs. the entire image file that is embedded?

Have the log use this:
11/19/08 10:33:19:187 - SENSIBLELOGOGREY_DATA:
11/19/08 10:33:19:187 - BO6BWMOVEEYES_DATA:
11/19/08 10:33:19:187 - OLDMANATPC_DATA:

vs. using this
11/19/08 10:33:19:187 - SENSIBLELOGOGREY_DATA:
11/19/08 10:33:19:468 - 424D36171100000000003600..........entire image file length..........
...
...


Is there a specific way to locate the imbedded images to avoid this? Right now I keep all my images in one area at the bottom like this:
Label>Imported BMP images
SENSIBLELOGOGREY_DATA:
424D36171100000000003600..........entire image file length..........
BO6BWMOVEEYES_DATA:
424DD2630000000000003000..........entire image file length..........
OLDMANATPC_DATA:
424D86A400000000000036040..........entire image file length..........
...
...
...

The file size gets compounded when the log files are also part of nightly backups that make more copies of the large log files.

[Marcus Tettmar]

I think your suggestion makes sense - don't output the actual data to the log file - just the name. I'll look into it.

[Bob Hansen]

Thanks Marcus.

I have also found another issue with these log files. I usually use TextPad for all of my editing, and it gets hung when trying to open these log files. I can open them OK in Wordpad, that is how I had to edit them, so this is a TextPad issue I guess. I wonder if it is because there is no EOF at the end of each of the image data files? Could be causing some confusion? But, if you can just use the reference, then this becomes solved also.

Thanks for listening.

[Marcus Tettmar]

No it's because TextPad can't handle the long line length. There IS an end of line at the end of the line but the line is *HUGE*. Most editors can't handle this. The old Macro Scheduler editor was about the only one that could. Every other editor we tried crashed or hung up simply because it could not cope with lines beyond a certain length.

This was an issue when we decided to replace the editor in v11. Unfortunately in order to add the functionality we wanted we could no longer support such long lines. Therefore you might notice that v11 imports binary data a bit differently - instead of one long line it splits the line up into many smaller lines. So you should find that TextPad can cope with NEWLY created V11 scripts. I say NEWLY because if you created your script in v10 and imported to v11 it will still have long lines for the binary data.

Create a new script in v11 and import some binary data, or reimport the binary data into an old script using v11 and you should find TextPad can cope.

[Marcus Tettmar]

Bob,

I've looked into this and realise it isn't so simple to make the logging ignore these lines as there is no way to be absolutely sure the line isn't something else. At the same time I realise there's a better, easier solution which you can already use. Place an Exit> statement BEFORE your image labels.

In fact v11 already does this automatically. When you import an image it places an Exit>0 statement before the imported data. This prevents the script ever executing beyond the Exit statement and trying to execute the image data and will therefore mean it won't get logged. The image labels are still usable of course as any reference to them causes the script to be searched for the label regardless of the Exit statement which just prevents execution itself continuing.

[Phil Pendlebury]

PSPAD seems to handle them OK. Just a fyi.

http://www.pspad.com/

[JRL]

Back in May we had a similar log file discussion. I asked then if there could possibly be a system variable to turn logging on and off from within the script.

Or perhaps a system flag
Let>WRITE_LOG_FILE=1
//do stuff that writes to the log
Let>WRITE_LOG_FILE=0
//do stuff that does not write to the log

Would something like that be workable?

[Bob Hansen]

So this would be a command in the script like this?

IMAGEDATA:
Let>WRITE_LOG_FILE=0
5309850385039509370280850398059809......
Let>Log=ONLet>WRITE_LOG_FILE=1
IMAGE2DATA:
Let>WRITE_LOG_FILE=0
43500470493503985....
Let>WRITE_LOG_FILE=0

Yes, that's I great idea. Now that you reminded me, I seem to remember agreeing with you when you first suggested that. That seems like it would be much easier for Marcus to implement also.

The EXIT>0 solution also sound like good temp fix for the imbedded files in the meantime. This was a script compiled in version 9 I believe, so I won't see the lines split up. No need to recompile for my client, but I may do it just to see and confirm the result. I am guessing that I cannot just copy and paste.....can do that for the code, but will need to do the imported image data again?

[JRL]

Would something like that be workable?

Update.... Not only workable but done!!!! In less than an hour!!!! Thank you.






.

[Marcus Tettmar]

Yes, if you copy and paste the image you'll just copy over one big line. So you'll want to delete that part and then re-import it with the new import option in the new editor if you'd rather have it split into shorter lines. Only v11 can cope with that, v9/v10 will only read the first line. V11 will read the data until a blank line is encountered.

Note that I have just announced beta build 025 which has a _WRITE_LOG_FILE variable.

[Bob Hansen]

Thanks to Dick (JRL) for the great suggestion, and this is an ideal time to remind everyone that the support at Macro Scheduler is second to none!

Marcus is available, open to suggestions, listens well, and responds immediately. I feel good about support for other companies also, but the MJT support for Macro Scheduler surpasses all of them.

Thanks again Marcus for your unique services.

[Phil Pendlebury]

the MJT support for Macro Scheduler surpasses all of them.

I'll drink to that Bob.

Indeed Marcus is a model for any of us supporting our own software too. I have learned so much from his attitude and delivery.

Marcus you should consider writing a book "How to deal with customers and keep them happy."

I would buy it.