June 19, 2014

Load Testing – Measuring Elapsed Time

Filed under: Automation,Macro Recorder,Testing — Marcus Tettmar @ 12:02 pm

In a support email recently I was asked how to record a macro to perform load testing.

While you could certainly use the macro recorder for much of the process, in most cases you are probably going to want to insert some code to make the macro more dynamic in waiting for events to complete. Your load testing is going to slow things down and what you want is to measure how long things take. So you’ll need to be sure you’re waiting for “readiness” and calculating how long that took.

The macro recorder by its very nature records the firing time between distinct events like mouse clicks and keystrokes exactly as it happened at play back. It does however insert “WaitWindowOpen” commands when new windows appear and these will be dynamic – they will wait however long is needed for windows to open and close.

But more often than not you’ll want other forms of waiting. My personal recommendation is to use WaitScreenImage to wait for a visual cue on the screen. This is closely analogous to how a human uses the application – she uses her eyes to see what’s on the screen and determine “readiness”.

If you’re load testing then you’re probably going to run lots of macros – or virtual users – at the same time and are going to slow things down. This is exactly what you want to do – you want to see what the effect of adding more users is – and you’ll need to measure elapsed time. So you’ll need the macro to wait exactly as long as necessary.

The easiest way to do that is with Image Recognition. By all means use the macro recorders to record the simple bits – the bits that send keystrokes into fields. Or use the UI functions and the handy Find Object Wizard.

Also use the Screen Image Recognition wizard to record your waits. It’s real easy. Just capture the thing on the screen you want to wait for and it will spit out the code to do that.

So once you’ve done that you have a macro which will carry out the process and wait for things to complete and be ready before continuing. Now you will want to gather statistics to see what the elapsed times are and compare with different scenarios.

You might do this simply by looking at the log file for the macro. The log file outputs the time each command executed. So you could calculate the elapsed time between starting a step and when it completed. I’ve known customers simply open the log files in Excel and add their own formulas.

But I would make use of the Timer function to have the script itself measure the elapsed time of specific events. You might then output this elapsed time to an Excel or CSV file or store it in a database. It’s up to you.

Here’s a simple bit of code which sets a timer, runs Notepad, waits for it’s main window and then calculates elapsed time, storing it in a CSV file.

//start the Timer
Timer>startTime

//Run Notepad as an example
Run>notepad.exe
//wait for it to be ready
WaitWindowOpen>Untitled - Notepad

//How long did that take?
Timer>endTime
Let>elapsed=endTime-startTime

//elapsed is in milliseconds - write it to our CSV file
WriteLn>c:\temp\test_1.csv,wRes,STEP1;%elapsed%

This is really just to show you how the Timer function allows you to measure elapsed time – you’ll likely have a bit more going on and if “readiness” is not signalled by a new window as it is here then WaitScreenImage might be a better option.

September 20, 2013

50% Off Software Test Automation eCourses

Filed under: Announcements,Automation,Testing — Marcus Tettmar @ 3:32 pm

Just a heads up that our friend Randy Rice is currently offering 50% off his most popular Software Test Automation eLearning courses.

So if you’re interested in learning about software test automation you can grab yourself a bargain until Midnight Tuesday. Randy’s got the info here.

May 25, 2013

Test Plan Automation 101 – Macro Scheduler Style

Filed under: Automation,General,Testing — Antonius @ 10:18 am

This is a guest post by Antonius Momac and Jana Krawiec, QA Automation Analysts at Triple Point Technology.

“If you don’t know where you’re going, any road will take you there”

– George Harrison

The main questions to ask when considering automated software testing are: “Where do I start?”  and “Where is it that I want to go?”  If you spend too much time on the wrong path, you’re either going to fail or overspend on time and resources.  Therefore it’s a good idea to get a clear picture of how information systems (applications under test) work.  I don’t mean at the low-level of zeros and ones that are crunched by the CPU, but rather where human meets machine.

The fundamental to any software process is that there are three main parts:

the Input (start), the process (main action), and the output (end) .  From the input, some data is taken and changed by the process which generates the output; i.e. modified data.

As an automation engineer, the choice of what part of the process to automate is yours. Automate as much or as little as you see fit.  Think about your particular situation and what makes sense.  Do you automate the setup?  Or is it better to automate the steps of the test plan (the driver/autopilot) and/or the verification?  Better yet, all three, or just one of them?

We started with a manual test plan and fully automated it.  Simply because the Macro Scheduler scripting language is flexible enough to support automation from all approaches. Automating the setup, the input, the process (say clicking on a button, or some mouse actions), as well as the verification of the output and reporting went smoothly.  We created small scripts that were managed from a master script.  A full and complex four hour test plan was built.  Everything worked great until we put it to the action testing a new build. We discovered that our GUI is in constant flux, which caused major changes to wreak havoc on our automated scripts.  Real time maintenance of scripts was required, thereby slowing down the entire testing process.  Ironically, this is the exact opposite of what test automation is all about.

In the end, full automation didn’t work for us. Less technical minded testers were not interested nor capable of learning how to debug a failing script.  Moreover, with the introduction of new features and constant GUI changes, our major focus during testing went to maintenance of our automation routines instead of finding bugs in the new release. Knowing that we picked the best possible software for automation allowed us to start looking for alternative approaches.  We decided to bypass the GUI interaction and focus on the verification process, since we never experienced issues with it.  Before long, we were getting through the testplan faster by manual GUI interaction combined with the verification process. It allowed us to test the application quickly, accurately and in depth.  Once we got the best of both worlds, we decided to develop a tool that would handle the verification with minimal setup.  With this tool anybody could set it up and get right into regression testing and finding bugs faster.  It allows us to integrate automation into new test plans very quickly.

Let’s analyze a test plan of 156 US-Letter size pages in length to get a better understanding.  The Majority of the test plan, 125 pages (82.6%), are dedicated to verification or what we call a checkpoint.  Further breakdown of those 125 pages revealed that there is 1 verification process is being repeated 50+ times consuming 108 pages. Manual verification is tedious and time consuming. Automating this one repetitive process eliminates 70%(108 pages) of the entire test plan in one shot.

Best of all removing, adding or updating automated routines proves to be an efficient process. No need to reconstruct or redesigning existing scripts and routines, saving us plenty of valuable time with minimal maintenance. Clearly, this is the road anybody wants to be on when it comes to automating your test plan.

So when you’re thinking about automating your test plan, consider these three rules of thumb:

  • What needs to be automated – Any fixed and repetitive processes where automation will save time, and ensure accuracy.  Big wins equal huge time saving with little code.

  • What can be automated – depends on the tool you’re using, since Macro Scheduler can be used to automate any process, be careful.  Automate, but automate wisely…  Your milage may vary.

  • What should never be automated – pieces that require excessive time and resources allocated disproportionately to the amount of time saved (mostly from maintenance point of view).  Maybe your test plan needs to be reworked.

Remember Bill Gates quote: “…automation applied to an efficient operation will magnify the efficiency. …automation applied to an inefficient operation will magnify the inefficiency.”

Contact the authors:

Antonius Momac Twitter handle: @MoMacAttack and remember “If you’re not creating TROUBLE, you’re not creating much!
Automation Diva Jana Krawiec Twitter handle: @krawiecjp

May 9, 2013

Test Plan Automation Through SQL Verification

Filed under: General,Testing — Marcus Tettmar @ 1:05 pm

Our good friend Antonius Momac just emailed me to tell me about this video he has put together on automating test plans with SQL verification:

Macro Scheduler users will probably recognise that Antonius is using Macro Scheduler to drive these tests [ keep an eye out for the Macro Scheduler icons and custom dialogs 🙂 ]

July 26, 2010

Practical Software Test Automation Course Now Available in e-Learning!

Filed under: Announcements,Testing — Marcus Tettmar @ 11:07 am

Rice Consulting have just announced that their new Practical Software Test Automation Course – which features Macro Scheduler for the scripting exercises – is now available in an e-Learning format.

This course focuses on the basics of software test automation and expands on those topics to learn some of the deeper issues of test automation. This course is not specific to any particular tool set but does include hands-on exercises using free and inexpensive test tools. The tool used for test automation exercises is Macro Scheduler.

The main objective of this course is to help you understand the landscape of software test automation and how to make test automation a reality in your organization. You will learn the top challenges of test automation and which approaches are the best ones for your situation, how to establish your own test automation organization, and how to design software with test automation in mind. You will also learn many of the lessons of test automation by performing exercises using sample test automation tools on sample applications.

I’ve been running through the course myself and so far I’m really impressed. This is a great way to get started on test automation and the great value e-Learning format means you can do it in your own time at your convenience. There’s also a whole module on using Macro Scheduler with some videos to help you get started.

You can see the course outline here, take a demo, or sign up on line here.

May 21, 2010

Weekly Forum Round-up

Filed under: Automation,General,Scripting,Testing — Marcus Tettmar @ 10:34 am

I thought I might start a weekly round up of some of the Macro Scheduler forum posts that caught my eye during the week. Not everyone gets a chance to browse the forums all the time, so it might help to link to some here. Then those that subscribe to the blog via RSS/Email will see them and they’ll also show up in Macro Scheduler’s News Feed window.

The forums are quite active but I won’t link to every little discussion or request for support – just those that demonstrate a new feature, or work as a “how to” or anything else that I think could be useful. We’ll see how it goes.

So here’s my list for week ending 21st May 2010:

How to trim spaces from the ends of, or within, a string

A Progress Bar Demo using Macro Scheduler 12

How to make a custom dialog minimize to the task bar

Putting a status bar with multiple panels on a custom dialog

Finding a drive based on its label (volume name)

Getting the HTML source of a page using WebRecorder

Running Automated Testing scripts in the background via VMWare

May 19, 2010

Test Automation Class Update; European Dates

Filed under: Announcements,Automation,Testing — Marcus Tettmar @ 10:06 am

Randy has just reported on his blog that the first run of his new Test Automation class which features Macro Scheduler in the hands-on exercises was a great success.

The next class will be held next month in Rome, Italy on 16/17 June, where Randy will also be presenting a workshop entitled “Innovative Software Testing Approaches“.

So if you’re this side of the pond and looking to improve your skills, with the added advantage of a business trip in a beautiful, historical city, check it out.

May 13, 2010

New Software Testing Course Featuring Macro Scheduler

Filed under: Testing — Marcus Tettmar @ 10:11 am

If you’re new to Automated Software Testing or looking to expand your knowledge you wouldn’t go far wrong with Randy Rices’ training courses.

Rice Consulting specialise in software testing training, certification and consulting. Their new Practical Software Test Automation course focuses on the basics of software test automation and expands on those topics to learn some of the deeper issues of test automation. The course includes hands on exercises using Macro Scheduler.

The course is aimed at Software Testers, Test Automators, Developers and Test Managers.

I’m told an e-learning version of the course is also in the works and I’ll let you know more when I hear about it.

May 11, 2009

Test Validation Techniques

Filed under: Automation,Testing — Marcus Tettmar @ 1:42 pm

We recently received the following query to support:

I’m interested in Macro Schedular for GUI testing. How do I verify that the test has succeeded or not? 

I thought it would be useful to post my response here:

There are a number of ways you could do this. Which one you use might depend on the type of application you are automating, or your specific requirements. You could:

  • Capture object text to see if it contains the data you would expect using such functions as: GetControlText, GetWindowText or Windows API functions. See: http://www.mjtnet.com/AutomatedTesting.pdf
  • Capture object and other screen text using the Screen Scraping functions: GetWindowTextEx, GetTextAtPoint, GetTextInRect. Compare captured text to expected data. There’s a sample script called Text Capture which you can use to test what text you can capture. Run it, and point the mouse cursor at the text you want to capture and confirm you can see it on the dialog. See Screen Scrape Text Capture Example and Screen Scraping with Macro Scheduler
  • Compare visually: Capture screen shots (or just windows) and compare the captured bitmaps with bitmaps captured at design time. Use the ScreenCapture and CompareBitmaps functions. This solution has the benefit of working with ANY technology on ANY platform. When you create the routine you capture the screens as they should appear when valid. So at runtime after entering data and controlling the app the macro would capture the screen/window and then compare to the valid images thus determining success or failure. See: How to use Image Recognition
  • There may be other options, especially for non-UI processes, such as reading data from the apps database (using DBQuery) or reading from a text file (ReadLn, ReadFile) or CSV file, checking the existence of files etc – depending on what the application you are testing does and what signifies a valid outcome.

Are you using Macro Scheduler for automated testing?  What types of app are you testing and what methods are you using?  Please comment below.

December 5, 2008

Automated Software Testing Success

Filed under: General,Testing — Marcus Tettmar @ 5:20 pm

A couple of great posts in the new Software Testing forum are demonstrating the power of Macro Scheduler for automated testing.

Antonius Momac says in this post:

I’ve looked at more than 8 other automation tools. Some were from the big companies with many products like IBM (rational), HP/Mercury, and other more focused shops like Automated QA, Tethy Solutions, Redstone (eggplant) ….. However, the application I was putting under test was very particular. In fact, plain and simple, Macro Scheduler was the only tool that worked.

In this post Gale Dyvig said:

We found the problem with object orientated test tools like WinRunner, QTP, SilkTest, etc is the partial or nonexistant support for the objects under test. As new technologies are created the problem gets worse.

These tools have image based workarounds, but they need the script to specify the image location on the screen. We tried some workarounds based on a WINAPI command similar to GetPixelColor to find the images. It worked but was nowhere as well developed or convenient to use as Macro Scheduler’s FindImagePos. If you have bmp’s the images your application will display, you can write a script before the application is coded.

Macro Scheduler and AppNav are great because the same solution works for all technologies runnable on Windows. Because the solution is image based it works in development, independent test, and production environments.

Are you using Macro Scheduler for automated software testing? If so please post an introduction in the Software Testing forum. I’d love to know more about how people are using Macro Scheduler in automated software testing scenarios.

Older Posts »