This is a guest post by Antonius Momac and Jana Krawiec, QA Automation Analysts at Triple Point Technology.
“If you don’t know where you’re going, any road will take you there”
– George Harrison
The main questions to ask when considering automated software testing are: “Where do I start?” and “Where is it that I want to go?” If you spend too much time on the wrong path, you’re either going to fail or overspend on time and resources. Therefore it’s a good idea to get a clear picture of how information systems (applications under test) work. I don’t mean at the low-level of zeros and ones that are crunched by the CPU, but rather where human meets machine.
The fundamental to any software process is that there are three main parts:
the Input (start), the process (main action), and the output (end) . From the input, some data is taken and changed by the process which generates the output; i.e. modified data.
As an automation engineer, the choice of what part of the process to automate is yours. Automate as much or as little as you see fit. Think about your particular situation and what makes sense. Do you automate the setup? Or is it better to automate the steps of the test plan (the driver/autopilot) and/or the verification? Better yet, all three, or just one of them?
We started with a manual test plan and fully automated it. Simply because the Macro Scheduler scripting language is flexible enough to support automation from all approaches. Automating the setup, the input, the process (say clicking on a button, or some mouse actions), as well as the verification of the output and reporting went smoothly. We created small scripts that were managed from a master script. A full and complex four hour test plan was built. Everything worked great until we put it to the action testing a new build. We discovered that our GUI is in constant flux, which caused major changes to wreak havoc on our automated scripts. Real time maintenance of scripts was required, thereby slowing down the entire testing process. Ironically, this is the exact opposite of what test automation is all about.
In the end, full automation didn’t work for us. Less technical minded testers were not interested nor capable of learning how to debug a failing script. Moreover, with the introduction of new features and constant GUI changes, our major focus during testing went to maintenance of our automation routines instead of finding bugs in the new release. Knowing that we picked the best possible software for automation allowed us to start looking for alternative approaches. We decided to bypass the GUI interaction and focus on the verification process, since we never experienced issues with it. Before long, we were getting through the testplan faster by manual GUI interaction combined with the verification process. It allowed us to test the application quickly, accurately and in depth. Once we got the best of both worlds, we decided to develop a tool that would handle the verification with minimal setup. With this tool anybody could set it up and get right into regression testing and finding bugs faster. It allows us to integrate automation into new test plans very quickly.
Let’s analyze a test plan of 156 US-Letter size pages in length to get a better understanding. The Majority of the test plan, 125 pages (82.6%), are dedicated to verification or what we call a checkpoint. Further breakdown of those 125 pages revealed that there is 1 verification process is being repeated 50+ times consuming 108 pages. Manual verification is tedious and time consuming. Automating this one repetitive process eliminates 70%(108 pages) of the entire test plan in one shot.
Best of all removing, adding or updating automated routines proves to be an efficient process. No need to reconstruct or redesigning existing scripts and routines, saving us plenty of valuable time with minimal maintenance. Clearly, this is the road anybody wants to be on when it comes to automating your test plan.
So when you’re thinking about automating your test plan, consider these three rules of thumb:
-
What needs to be automated – Any fixed and repetitive processes where automation will save time, and ensure accuracy. Big wins equal huge time saving with little code.
-
What can be automated – depends on the tool you’re using, since Macro Scheduler can be used to automate any process, be careful. Automate, but automate wisely… Your milage may vary.
-
What should never be automated – pieces that require excessive time and resources allocated disproportionately to the amount of time saved (mostly from maintenance point of view). Maybe your test plan needs to be reworked.
Remember Bill Gates quote: “…automation applied to an efficient operation will magnify the efficiency. …automation applied to an inefficient operation will magnify the inefficiency.”
Contact the authors:
