Searching huge text file - solution!

Technical support and scripting issues

Moderators: Dorian (MJT support), JRL

Post Reply
User avatar
Dorian (MJT support)
Automation Wizard
Posts: 1348
Joined: Sun Nov 03, 2002 3:19 am
Contact:

Searching huge text file - solution!

Post by Dorian (MJT support) » Wed Nov 20, 2002 6:55 pm

Wohooo! I just figured out how to search large text files quickly. i just tested this script and it completed a search on a 176MB text file in only a second or so.

_____________________________

let>filename=type c:\testfile.txt | find "searchtext" /i > c:\testfile2.txt
WriteLn>c:\testfile.bat,result,%filename%


Let>RP_WINDOWMODE=0
Run Program>c:\testfile.bat

REMARK>I had to enter a wait command because
REMARK>this script seems to search huge files
REMARK>so fast it will try to open the output
REMARK>file before windows can generate it.
REMARK>If anyone finds a better way of doing
REMARK>this please let me know.

message>Finished Search, generating output file
wait>5
executefile>c:\testfile2.txt

User avatar
Dorian (MJT support)
Automation Wizard
Posts: 1348
Joined: Sun Nov 03, 2002 3:19 am
Contact:

Post by Dorian (MJT support) » Wed Nov 20, 2002 6:56 pm

One question - how do I get around the space problem in a path when using a .bat file? for instance, "c:\testfile.txt" is fine, but "c:\my document\testfile.txt" won't work. I know there's probably a very simple answer.

Ernest

Post by Ernest » Wed Nov 20, 2002 11:49 pm

Hi,
congratulations to that "oneliner" :-)

Cause there's nothing like "RunWait" (stops execution of the script till program has finished) which I know from AutoIt, I've helped myself with this:

Code: Select all

Label>FileProgressCheck
FileSize>c:\testfile2.txt , fsize1
Wait>0.5
FileSize>c:\testfile2.txt , fsize2
If>fsize2>fsize1, FileProgressCheck

Since WIN9x,

START "c:\my document\testfile.txt"

should work with "space" lines in batch files.


Rgds,
Ernest

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

RunWait and Spaces in path

Post by Marcus Tettmar » Thu Nov 21, 2002 12:24 am

Hi,

In version 7.1 there *is* something like RunWait. Do this:

Let>RP_WAIT=1
Run>yourbatfile.bat

I often wonder if anyone ever reads the history list or the list of new features in our update announcements, and often wonder how many new features go completely unnoticed ;-)

See the help file topic for Run Program. Note that this is new to version 7.1

Regarding spaces in the filename, simply use quotes, as you would anywhere else in Windows:

Run>"c:\some directory with spaces\some place\mybatfile.bat"
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

User avatar
Dorian (MJT support)
Automation Wizard
Posts: 1348
Joined: Sun Nov 03, 2002 3:19 am
Contact:

Post by Dorian (MJT support) » Thu Nov 21, 2002 3:33 am

Thanks guys.

Those "'s work perfectly. I amended the search line to :

let>filename=type "c:\my documents\testfile.txt" | find "test" /i > "c:\my documents\testfile2.txt"

and now it allows me to search a file if there's a space in the path. :D

Then I added :

DeleteFile>c:\testfile.bat

... at the end, because each time the script was run it added another line to the batch file and it ended up searching the textfile four or five times.

I cannot believe how fast this searches such a large file.

So now I know this can be done easily I'll have to make the script do something actually useful.

Ernest

Post by Ernest » Thu Nov 21, 2002 9:22 am

Hi H2000,

">" will create a new file and pipes the output to it. So you'll get always the result of the last execution. Nothing more. Therefore you don't have to delete it with every session. When you've finished completely, get rid of it ...

Code: Select all

//cmd = NT, command = W9x/...(?)
Run Program>cmd /c find "this" on\that\file.txt > output\file.txt
">>" will append output to the same file. You'll get the result of previous runs as well, piped to the same file (I'm using it for 24/7 logs).

Code: Select all

//cmd = NT, command = W9x/...(?)
Run Program>cmd /c find "this" on\that\file.txt >> output\file.txt
Ciao
Ernest

BTW: Marcus,
I won't get the new release (I'm blocked by "BeanCounters" at the finance department) :(

I's the current command reference (7.1) seperately available for download ? So I won't be endangered to advise "historic" ways of MSched coding ... :wink:

User avatar
Marcus Tettmar
Site Admin
Posts: 7378
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Post by Marcus Tettmar » Thu Nov 21, 2002 10:41 am

Ernest wrote:Hi H2000,

I's the current command reference (7.1) seperately available for download ? So I won't be endangered to advise "historic" ways of MSched coding ... :wink:
Yes, as always it is available for viewing from the support pages and available for download from the download page:

http://www.mjtnet.com/support.htm
http://www.mjtnet.com/downloads.htm
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

User avatar
Dorian (MJT support)
Automation Wizard
Posts: 1348
Joined: Sun Nov 03, 2002 3:19 am
Contact:

Post by Dorian (MJT support) » Thu Nov 21, 2002 6:21 pm

Hi Ernest, many thanks for the pointers.

I added the delete statements as this macro is just the beginning of a much larger one and my needs dicate the files are not appended. The output file is just a temporary file for the macro to run a few more checks on.

The "real" version will allow the user to input the filename and search term, which then uses concat and writes the new batch file, and i didn't want my macro to generate huge obsolete textfiles on their computer.

Ernest

Post by Ernest » Sun Nov 24, 2002 3:34 pm

Hi H2000,
Is it necessary to use a batch?
Why not run the scan directly using a command shell (the DOSBox)?

Code: Select all


If>%OS_VER%=NT.4.*,WNT
If>%OS_VER%=<CodeOfWIN2K>,WNT
If>%OS_VER%=<CodeOfWIN95>,W9x
If>%OS_VER%=<CodeOfWIN98>,W9x
Goto>Exit

Label>WNT
Let>ComIntpr=cmd.exe
Goto>ScanSet

Label>W9x
Let>ComIntpr=command.exe

Label>ScanSet
Let>scanfile=%SYS_DIR%\%ComIntpr% /c find "<searchpattern>" "c:\my documents\testfile.txt" > c:\testfile2.txt

Let>RP_WINDOWMODE=0
Run Program>%scanfile%

Label>Exit
Ciao
Ernest

User avatar
Bob Hansen
Automation Wizard
Posts: 2475
Joined: Tue Sep 24, 2002 3:47 am
Location: Salem, New Hampshire, US
Contact:

Post by Bob Hansen » Mon Nov 25, 2002 7:21 am

Hello Ernest.

I believe there is an error in the last script:
Label>W9x
Let>ComIntpr=command.exe
====================================

I think that "command.exe" should be "command.com". Correct line should be:
Label>W9x
Let>ComIntpr=command.com

But I like the approach to allow a script run on any Windows system. That's neat. Thanks for the new tool.

Ernest

Post by Ernest » Mon Nov 25, 2002 7:55 am

Hi,
Ooops, you're right (I'm used to run it without any file extension)!

I don't know if MSched 7.1 cares about the DOS SysVar "COMSPEC".
If that's the case, you don't have to predefine diff. CmdIntpr. sections!

E.

User avatar
Dorian (MJT support)
Automation Wizard
Posts: 1348
Joined: Sun Nov 03, 2002 3:19 am
Contact:

Post by Dorian (MJT support) » Tue Nov 26, 2002 5:40 am

Hi Ernest, that's really helpful, thank you. Would that work with XP too, or what would the line for XP be?

Ernest

Post by Ernest » Tue Nov 26, 2002 7:59 am

Hi H2K, :wink:
I guess - but sorry, I'm currently not using XP! Run SET at the commandline and check for the variable COMSPEC. Here you should? find the path to the command interpreter (no idea what XP offers in case of DOS functionality). I expect MSoft hasn't changed it. Should be "cmd.exe". Please check ...

BTW: You can do the same for WinME.

E. :)

User avatar
Dorian (MJT support)
Automation Wizard
Posts: 1348
Joined: Sun Nov 03, 2002 3:19 am
Contact:

Post by Dorian (MJT support) » Wed Nov 27, 2002 9:13 am

hi Ernest, FYI I tried "SET" in XP and this is what was returned under COMSPEC :

c:\windows\system32\cmd.exe

Over the weekend I bought a really useful book to help remind me of dos commands. It's called "Windows User's guide do DOS" by Carolyn Z Gillay & Bette A Peat.

It's an oldie but a goodie.

I already found I can add SORT to the FIND command, so it's proved realy helpful already. :D

Post Reply
Sign up to our newsletter for free automation tips, tricks & discounts