Bloggo back to the blog
Implementing Test Automation in Your Organisation; Getting Started With Automation-->
Test automation sounds so promising! But in reality, the high hopes it raises often lead to frustration and disappointment. This article explains how to go about implementing Test Automation and gives you guidelines for successfully implementing and maintaining automated testing so you can realise the promise while avoiding some of the pitfalls. You’ll find out about a number of issues that you’ll want to address before you introduce automation, and learn what you can do to successfully deploy and maintain automated testing in your organisation.
Analyse and Plan Before Introducing Test Automation
As with any big change, the first step in introducing test automation is conducting a situational analysis in order to identify needs across the organisation and consolidate them into a prioritised list. The results of the analysis will strongly influence your choices of alternative ways forward. To start, you need to answer some fundamental questions:
- Who are the stakeholders?
- What are their expectations?
- What are their business goals?
- What timeline do you need to follow?
- What tools do you have access to?
With answers to these questions, you’ll be able to make recommendations for an action plan. In order to anchor your plan with management, you may have to calculate the payback time on the investment of time and money you’ll need to get test automation up and running. Since the costs can be significant, it’s important that everyone is aware of the required effort from the outset. You’ll need to decide early on what tests you will continue to perform manually, and which ones will be automated. Even when you’re specifying requirements, you should define how tests would be performed at every level (component, integration, system and acceptance testing).
It will take perseverance before you begin to see the benefits of automated testing. Even after you’ve established a test automation program, your automation platform will require maintenance in order to remain up to date.
Choose test cases to be automated
Experts have debated for years which kind of test cases best lend themselves to automation. You can use several basic approaches to evaluate which test cases are worth automating, by considering the following factors:
- The time saved compared to manual testing
- Number of times the test case needs to be performed
- Number of variations in input data
- Whether you have well-known, manual tests
- Simplicity, established input data, and expected results
- Criticality/importance of the test case
- Whether the test case can be reused
- Frequency/propensity of change
- Stability of the system being tested.
Beyond the obvious: Automate other testing tasks
When looking at what you should automate, elevate your perspective and take in the big picture of testing work your team needs to perform. You’ll find many areas suitable for automation beyond the most obvious ones, such as regression testing. I find that teams often overlook tasks like preparation of test data, reporting, and configuration – all of which highly contribute to successful test automation. By integrating test management and your automation framework, you can save time passing results of test execution to your test administration tools.
Taking action: How to put automation in place
The process of implementing automation can be roughly divided into the following activities:
- Recording scripts
- Writing scripts
- Rewriting/customizing test cases
- Customizing test data
- Customizing and integrating test tools
- Integrating testware, e.g. test cases, test plans, test reports, and test tools.
Keep your automated tests up to date
Maintaining test cases is an important part of the test automation process. Before adding new test cases, you need to analyse what each test case would provide for the overall system, and how much effort it would take to maintain. One success factor is making sure every test case provides really detailed information about where in the code the bug resides. If you have to spend a lot of time analysing the results, it’s only going to increase your workload and make life harder.
Remember that you need to maintain test data to keep it relevant, and having a large amount of test data will do more than just increase the time it takes to keep it up to date. The handling itself will also take more time, for example, if you have to replicate a large database.
Depending on the test framework you choose, you may also have to maintain the framework itself, as well your internal libraries of shared functions.
Choose the right tools
Your choice of tools depends on your needs and budget, and of course on what you expect to get out of automation. There is a variety of tools on the market, ranging from comprehensive tools that provide support for large volumes of data in large organisations, as well as others that are open source, and specialised for a particular niche of the automation field.
It’s a good idea to consider how your choice of tools will affect the development process, since introducing tools often leads to more structure and standardisation. In an organisation where the work is relatively unstructured at the moment, the formalism and structure will likely be a welcome improvement. But if the organisation already has a well-functioning process, introducing a new structure can be an obstacle or a limitation.
Introduce chosen tools to the team
There are a few common “truths” about how to introduce new tools to the team. I recommend that you involve the people performing today’s manual tests early on, in order to gain their trust and support for automation within the QA organisation. That way you’ll help the testers to see introducing automation as a personal challenge that will ultimately benefit them. For example, the testers would be relieved of some drudgery and free to concentrate on more creative testing work. It is wise to plan manual and automated testing together, so as to get tan optimal mix between manual and automated test cases. Also, remember that test automation doesn’t replace manual testing – the two are complementary. Start small and let the team increase the automated portion gradually, and schedule maintenance time to ensure that automated test cases always stay up to date.
Use metrics effectively
Even before you launch automation, you should establish testing metrics so you have a basis for comparison from the outset. Questions that the metrics should be able to answer include:
- Time required for execution, compared with manual testing (including time for analysis).
- Test coverage for automated test cases compared to manual ones.
- Cost of automation compared with manual testing alone.
- Percentage of test cases that can be automated.
- Number of defects found that can only be found through automation (such as problems related to response times or performance).
Many metrics can also be automated, which means that some aspects of reporting will be easier too. Examples of metrics suitable for automation include:
- Code coverage
- Defect Detection Percentage.
Since metrics have wider implications, I recommend that you read our other articles that focus on metrics (see “Next steps” below).
It’s not cost-effective to automate everything. Tests you conduct frequently and that take significant time to perform manually are usually the best candidates for automation, so regression testing is usually a good place to start. At the same time, it can be difficult to calculate ROI (return on investment) on automation, since many factors affect your estimates, like how much time you can save on regression testing and how much time the team currently spends on maintenance. Manual regression tests are often executed sloppily, while automated testing is consistent, so you can achieve higher quality results by using the automated tests (but could have a hard time cost-justifying it). Here are some examples of ways you can calculate ROI:
- Cost-based – Calculate the cost savings achieved by automating a test suite compared to performing the tests manually.
- Time-based – Calculate the time saved by automated execution compared to the time needed to perform the same test suite manually.
- Quality-based – Calculate possible savings in terms of increased test coverage and reduced risk of errors in production.
Of these methods, I recommend the quality-based approach. The other estimates often lead to unrealistic expectations, ranging from the merely difficult to the impossible to achieve.
About the author
Ulf Eriksson is one of the founders of ReQtest, an online bug tracking software hand-built and developed in Sweden. ReQtest is the culmination of Ulf’s decades of work in development and testing. Ulf is a huge fan of Agile and counts himself as an early adopter of the philosophy, which he has abided to for a number of years in his professional life as well as in private.
Ulf’s goal is to life easier for everyone involved in testing and requirements management, and he works towards this goal in his role of Product Owner at ReQtest, where he strives to make ReQtest easy and logical for anyone to use, regardless of their technical knowledge or lack thereof.
The author of a number of white papers and articles, mostly on the world of software testing, Ulf is also slaving over a book, which will be compendium of his experiences in the industry. Ulf lives in Stockholm, Sweden.
Twitter – @ulf_reqtest