Grabbing variables in a text file.

Anthony · Post by **Anthony** » Tue May 31, 2005 6:15 pm

I have logs that are created with a lot of different information. I cannot change the way these logs capture information, but was wondering if there is a way to grab specific text after a specific keyword.

The log may look something like this:

192.168.10.10 - - [26/May/2005:13:11:41 -0400] "GET broadcast/1234567.rm?1=Anthony&2=Company Name, LLP RTSPT/1.0" 200 3147600 [WinNT_5.1_6.0.12.1069_RealPlayer_RN30BB_en-us_UNK] [97bd4e57-9c65-4151-a7bb-e2efe1972441] [Stat1: 5244 0 0 0 0 20_Kbps_Music_-_RealAudio][Stat2: 20671 20656 0 0 0 0 0 0 0 1 813 20_Kbps_Music_-_RealAudio] 0 0 1218 0 0 77027

Now I would basically want to grab specific information from this and paste it somewhere like an Excel spreadsheet or atleast write the information in a text file seperated by some common character so I could have a Text file that reads 1234567|Anthony|Company Name, LLP|1218 so I could load the text file in Excel and delimit by the specific character |

I would want to grab the information after "broadcast/" until ".rm" to grab all entries containing that number. I would also want to grab the information after "1=" and "2=" for name and company and also the number 1218 (this number varies) towards the end of the file after "RealAudio] 0 0 "

Also, there may be a lot of other numbers in many different lines after broadcast that I do not want. I would only want the information in strings containing the broadcast number for all entries matching 1234567 and the information stated above for that line.

I'm not sure if this is even possible. I could have it read the whole line and maybe use MidStr to grab some of the information, however, the length of the information varies. One entry could be 300 characters, and another could be up to 500. So thats why I was wondering if it would be possible to have it find "broadcast/" and get the next 7 characters, or grab from 1 location until a specific character.

Any help would be great!

*Edited for clarification, and pasted more accurate log entry.

Bob Hansen · Post by **Bob Hansen** » Tue May 31, 2005 8:13 pm

You should be able to do this using Position> and MidStr> to create some variables.
Or you could use VBScript with Regular Exression function.

I suspect that Position and MidStr may be easier.
Example you could use Position to get start value of "RealAudio] " then use MidStr with that value plus 5 to capture the "1218"

Code: Select all

//Read each line in the file, save as ThisLine

Postion>RealAudio] ,%ThisLine,%,LastBlock,0
Add>LastBlock,5
MidStr>%ThisLine%,%LastBlock%,4,LastValue
Message>The last value is %LastValue%

After you have the variables created for each line, you can then concatenate them with "|" character, and then do a WriteLn or a DDE to send the data to your other program.

Anthony · Post by **Anthony** » Tue May 31, 2005 8:30 pm

Thanks for the reply. I think I'm just confused on exactly how to use the commands...

I'm almost there, but there is just something I'm not grasping. So, I can use Position to grab the always 7 digit number after "broadcast/", but what would I use to grab something that could be any random length of characters up to a specific character such as the Name and Company example? None of these characters will ever be in the same location due to the IP that is logged first and other various information throughout the log.

I'm pretty sure you answered it in the example, but I hate to cut and paste someones knowledge without understanding exactly how it works.

Mainly I don't understand what's going on with the %'s

That might be a part of this program I've just never utilized before and not sure what it is I should be doing. I learn mostly from the command reference and there isn't too much detail on the Position/MidStr/Length commands.

I really appreciate any time and effort you have already spent trying to educate me... and would welcome more with open arms

Bob Hansen · Post by **Bob Hansen** » Tue May 31, 2005 11:36 pm

The % symbol is surrounding a variable that has already been defined. By using the %symbol, I am able to actually see the value of the variable in my log files. They are not always needed. See HELP on VAREXPLICIT.

I just used the last value in my example to show that you can use the Position of some other character(s) as an indicator to count from, so you do not need to know the exact position due to differing lengths. See HELP and examples on POSITION.

From your example:

192.168.10.10 - - [26/May/2005:13:11:41 -0400] "GET broadcast/1234567.rm?1=Anthony&2=Company Name, LLP RTSPT/1.0" 200 3147600 [WinNT_5.1_6.0.12.1069_RealPlayer_RN30BB_en-us_UNK] [97bd4e57-9c65-4151-a7bb-e2efe1972441] [Stat1: 5244 0 0 0 0 20_Kbps_Music_-_RealAudio][Stat2: 20671 20656 0 0 0 0 0 0 0 1 813 20_Kbps_Music_-_RealAudio] 0 0 1218 0 0 77027

we can see that 1234567 may be in a different position because the IP and/or date may cause different lengths. But it will always be right after "broadcast/". So we can use the position of "broadcast/" and add 10 characters to get the starting position to use in the MidStr> function.

Code: Select all

Let>SourceFile=Path\folder\name
Let>NewFile=Path\folder\name
//Start a loop doing ReadLn and do the following for each line.

ReadLn>%Sourcefile%,%LineNumber%,ThisLine
Position>broadcast/,%ThisLine%,1,Place1,0
Add>Place1,10
MidStr>%ThisLine%,%Place1%,7,Value1

//Continue in loop reading each value, maybe 4 different values?
//When done with Value1-Value6 then concatenate like this:

Let>NewLine=%Value1%|%Value2%|%Value3%|%Value4%
WriteLn>%NewFile%,result,%NewLine%

//Then continue in loop until all line are processed

See HELP on ReadLn and looping for LineNumber and example of loop "start" to make your own count loops for ? values and ? lines in file.
See HELP on Position and MidStr to see how values are used there.

Plenty of HELP and samples available. Put some code together and single step through the code with WatchList visible to see results of variables. Submit code and logs here so we can help modify it. Don't be afraid to copy/paste. Doing it is the best way to learn it. Be sure to remove Trailing Spaces from each line.

Anthony · Post by **Anthony** » Wed Jun 01, 2005 1:55 pm

Ok, I think I understand a little better. I was using an older version of Macro Scheduler and there seems to be more information in the newer help file.

It's still not working for me and I'm getting an Error in SubString Command.

Label>Begin
Let>VAREXPLICIT=1
Let>LineNumber=1
ReadLn>c:\test.txt,%LineNumber%,ThisLine
Position>broadcast/,%ThisLine%,1,Loc1,0
Let>Loc1=Loc1+10
MidStr>%ThisLine%,%Loc1%,7,RefNo
Message>This is the ref %RefNo%

Also when I try to message variable it shows up as "%RefNo%" Is that because I'm getting an error during the MidStr whenit's trying to write a value to the RefNo variable?

Anthony · Post by **Anthony** » Wed Jun 01, 2005 2:30 pm

Ok, so in the above example instead of using Add I used Let. The reason was because I was running into an error and forgot to change it back. When I use Add it seems like it thinks it's the date. and the MidStr is trying to grab characters starting at 1/9/1900 instead of Loc+10. Even if I try to use Let>Loc=Loc+10, it doesn't seem to be adding!

Just some more info on my errors if it helps anyone

Anthony · Post by **Anthony** » Wed Jun 01, 2005 2:59 pm

Ok so it was myfault for not reading up on VAREXPLICIT enough. I got that part down, I'm set now.

I just have 1 question remaining and hopefully I can answer it myself but I'm posting here incase because the help here has been great

I know I can find a specific location and grab a set number of characters, but now I'm wondering what do I do to grab characters of a random length? I know you can use the Length command but what I'm not putting together is if I'm trying to grab a specific variable, lets say someones name, or the duration in seconds, it could be any length. I know how to grab from a specific location up to a specific number of characters, but can I grab up to a specified character?

192.168.10.10 - - [26/May/2005:13:11:41 -0400] "GET broadcast/1234567.rm?1=Anthony&2=Company Name, LLP RTSPT/1.0" 200 3147600 [WinNT_5.1_6.0.12.1069_RealPlayer_RN30BB_en-us_UNK] [97bd4e57-9c65-4151-a7bb-e2efe1972441] [Stat1: 5244 0 0 0 0 20_Kbps_Music_-_RealAudio][Stat2: 20671 20656 0 0 0 0 0 0 0 1 813 20_Kbps_Music_-_RealAudio] 0 0 1218 0 0 77027

So I know I can grab the position after rm?1= but can I tell it to grab up to &2? and the numbers at the end, that 1218, could be a 3 digit number as well... any ideas?

Me_again · Post by **Me_again** » Wed Jun 01, 2005 3:43 pm

You are on the right track. Basically you have to also find the position of whatever follows the random length field and then you can calculate the length of the field from the two position numbers that you have.
Something like:

Position>broadcast/,%ThisLine%,1,Loc1,0
Position>whatever,%ThisLine%,1,Loc2,0
Let>fieldlen=Loc2-Loc1

Anthony · Post by **Anthony** » Wed Jun 01, 2005 3:49 pm

Ahhh yeah ok that makes sense! Can't believe I didn't think of that

Thanks for the advice!

Anthony · Post by **Anthony** » Wed Jun 01, 2005 7:41 pm

Thanks everyone who helped me with this! I got it working... kinda

The problem I'm running into is grabbing the Duration at the end of the line.

192.168.10.10 - - [26/May/2005:13:11:41 -0400] "GET broadcast/1234567.rm?1=Anthony&2=Company Name, LLP RTSPT/1.0" 200 3147600 [WinNT_5.1_6.0.12.1069_RealPlayer_RN30BB_en-us_UNK] [97bd4e57-9c65-4151-a7bb-e2efe1972441] [Stat1: 5244 0 0 0 0 20_Kbps_Music_-_RealAudio][Stat2: 20671 20656 0 0 0 0 0 0 0 1 813 20_Kbps_Music_-_RealAudio] 0 0 1218 0 0 77027

I basically need to grab the last number after the last "] 0 0 " I'm able to grab the number, and it works fine as long as the number is 3 or 4 digits. However, anything less than that, lets say it was 9, would grab "9 0 " I don't know how to get it to stop at the first space.

Anyone got any ideas?

Me_again · Post by **Me_again** » Wed Jun 01, 2005 8:59 pm

At a quick glance what I think I would do is grab the whole numeric tail of the string " 0 0 1218 0 0 77027" which is then a space delimited list of values - so you can use the "Separate" command to extract the nth (3rd in this case) number from the string.

Anthony · Post by **Anthony** » Thu Jun 02, 2005 2:29 pm

Ahhh.. worked perfectly!

One last thing, I finally went in and put in IF in for when it's done reading the file, and everytime it gets to the end, I get the following error:

Access violation at address 004CA5EC in module 'msched.exe'. Read of address 00000000.

ReadLn>c:\Documents and settings\%USER_NAME%\Desktop\%origref%.txt,%LineNumber%,ThisLine
If>%ThisLine%=##EOF##,ExcelFormat
Position>broadcast/,%ThisLine%,1,RefLoc
Add>RefLoc,10
MidStr>%ThisLine%,RefLoc,7,RefNo
If>origref=Refno,DisconnectPull
IF>origrefRefno,NextRead

I've used this feature before without any problem, just wondering if I did something wrong.

I'm on the last steps of getting this baby working exactly how I need it, and now something simple isn't working

Me_again · Post by **Me_again** » Sat Jul 02, 2005 3:09 am

In the above examples a "0" is used for the "relative" parameter for Position (Position>substring,string,start,result[,relative] )

Code: Select all

Position>broadcast/,%ThisLine%,1,Place1,0

I've been trying to get absolute position with a start position > 1 and can't make 1/0 work so I believe it needs to be spelled out as TRUE or FALSE instead.

Code: Select all

Let>VAREXPLICIT=1
Let>mystring=abcdefghijklmno
Position>k,%mystring%,1,kpos_0
Position>k,%mystring%,3,kpos_1,TRUE
Position>k,%mystring%,3,kpos_2,FALSE
MessageModal>%kpos_0% %kpos_1% %kpos_2%

Returns 11 9 11 which is correct.

Code: Select all

Let>VAREXPLICIT=1
Let>mystring=abcdefghijklmno
Position>k,%mystring%,1,kpos_0
Position>k,%mystring%,3,kpos_1,0
Position>k,%mystring%,3,kpos_2,1
MessageModal>%kpos_0% %kpos_1% %kpos_2%

Returns 11 9 9