Test Automation

Introduction

Getting started with test automation can seem daunting. How do you know where to start and what to focus on? We’re getting actionable and concrete advice from a Selenium enthusiast and UI testing whiz Diego Molina at Sauce Labs:

“I currently work as a software engineer at Sauce Labs. I co-created Zalenium, and help maintain the most used docker-selenium images. But it might be helpful to share with you some background on my personal testing story. I too was once just getting started with automation, and hopefully my experience can help you do the same.”

Step 1. Getting Started with test Automation – understand the benefits

Years ago, after getting my degree in computer science, I got a job as a developer. At that time, testing was primarily done by end users—we would hand over spreadsheets with all the test cases. Later, I got a job as an automation engineer where I wrote tests for every single use case. I started doing UI automation with Selenium and took the same approach, again writing tests for every use case.

At the time, I didn’t think about whether that was the right approach or not. But then I noticed a problem. My test suite went from taking 20 minutes to run to upwards of 60-70 minutes! Uh-oh. Could it be too many tests? Or did I just need to tune the infrastructure?

I ended up building my own infrastructure and scaled it up. I was able to improve the performance of the tests back down to 15-20 minutes. So then, of course, I thought the best idea would be to—you guessed it—add yet more tests. I started writing tests for every use case again.

It’s a vicious cycle! After a few rounds of this, I realized I was not tackling the problem the right way, and that automation was the key to solving it.

Why Automate?

 

Simply put, automation gives you the opportunity to spend less time on repetitive tasks and move on to solving bigger challenges, hopefully ones that are more fun.

Also, there is a trend in the software industry: getting software out the door faster and faster. We are finding ways to speed up our releases more and more, and in order to keep up, we have to automate.

 

 

I’ll cover 5 specific areas that will help you jumpstart test automation and get the most out of it in 5 easy steps

• Team Setup: I’ll give advice on how to handle various team setups and ensure great communication, whether the testing team is integrated with the dev team or stands alone.
• Testing Framework: I’ll discuss why to use a framework, talk about what features a framework should have—and why you should consider open source.
• Test The Right Thing: We can’t automate everything. Creating and maintaining automated tests takes time, so I’ll help you think critically about what to automate.
• When To Run Tests: I’ll discuss the evolution of testing and how to use automation to move your organization forward toward Continuous Testing.
• Get People to Join You: In the final installment, I’ll offer advice on how to motivate yourself, your entire team, and your company to invest in automation (and discuss the risks of not doing so).

 

Team Setup

 

When testers are on a separate team to developers

Choose a programming language such as Ruby or Python that has an easy setup and low barrier to entry. In fact, according to the 2019 Stack Overflow survey, Python has recently proven to be one of the most popular languages. But what’s more important than programming language? One word: communication. You have to communicate regularly with the product owner and with the development team in order to test effectively.

Here are some specific action items to facilitate this communication:

• Get access to the story/task/feature definitions to create tests.
• Get business context from the product owner to prioritize testing activities.
• Get as much information as possible about the tech stack, system architecture, etc.

Also, be in sync with the dev team. Go step by step as you’re implementing the first automated tests. Work together to agree on a working pace and define a bug report flow.

When testers and developers are on the same team

This can look one of two ways: either the testers and developers have separate, defined roles (QA/Dev), or each team members serves both functions as part of their role. It really helps when one or more team members are responsible to guide the team in testing.

In this scenario, it also helps when application code and test code use the same programming language, as it really leverages team collaboration. Also, when you’re testing, take advantage of the good practices used in the software development lifecycle. Keep test code in the same repository as the app code. This simplifies the flow required to write tests for features that are not yet in production.

The bottom line is that you should look at test automation as a software project… because it is!

Key Takeaways

• Invest time looking for a test framework that helps the team move faster. I will cover this in more detail in my next post, but this can be a huge time saver and make everyone more efficient.
• Make test design and implementation a team task. This cannot just be one person’s job: testing and quality need to be a part of the overall team culture.
• Document how tests are structured.Keep records of configuration as well as how to run, add and remove tests.
• Pair program and do code reviews.When you’re adding new tests, it’s really helpful to pair up with a teammate to check each other’s code and think through each other’s approach.
• Continuously evaluate if the existing test cases deliver enough value. We always want to be sure that we are testing the right thing and getting the results we need. Constantly evaluate and make sure this is happening!

 

How the team setup should evolve over time

No matter what your team setup looks like, there’s always room to evolve over time to get better and better. The goal is constant improvement, moving toward more efficient practices to let automated testing do the heavy lifting for you.

Of course, there will always be room for manual testing. However, the focus on it will change: manual testing will be used strategically for debugging and exploratory testing, instead of dominating your testing landscape.

Step 3. Testing Framework

What is a testing framework and why should you have one?

A framework isn’t just one thing. It’s a union of different things—guidelines, coding standards, good practices, tools and libraries—that work together to make us faster and more efficient. An effective test framework can lower maintenance costs, improve code reusability, help us focus on writing tests instead of writing boilerplate code before coding tests, and more. The point is that a single/library tool will hardly cover all my testing needs, but putting together a few testing tools and libraries that work in harmony will help me get there. When I say “framework,” that’s what I’m talking about.

From my perspective, there are three reasons to use a framework:

• Reduce maintenance costs– a framework simplifies sharing common functionality that’s needed across the team.
• Be faster– we can make changes by only changing things in one place when we have a single testing framework.
• Be reactive– we can bootstrap code to write tests, and this is embedded in the framework. When we are creating new test cases, we can get to implementation much faster.

What features should your framework have?

The below list is a guideline, not necessarily a set blueprint—and there may be additional things to consider given your individual situation.

• Assertions on actions
• Initialization and cleanup
• Data modelling/mocking configuration
• Site modelling abstractions
• Wrappers and helpers
• API usage
• Built-in reporting or easy to plug in
• future ready features
• Speed
• Debugging features
• Cross Browser
• Simulators-Emulators/Real Devices

 

It’s also important that you take into account whether the framework you’re considering is future-ready. Does it change and adapt to trends, which will enable you to evolve appropriately?  For example, will it allow the dev team to switch JavaScript frameworks?

Consider this scenario. If your web app is written in Angular, but the team decides for some reason to change it to React, will your framework allow you to start testing the React app without too much additional work?

Another good example why you need a “future ready” framework could be a situation where the team starts out only needing to write tests for a web application, but after some time has passed, they develop a mobile app that needs to be tested via Appium as well.

“An effective test framework should be able to support/leverage the team to write tests for the mobile app without the need of going through all the hassle of setting everything up from scratch.”

How to build your testing framework

Two words: open source. “But Diego,” you may say, “Why should I use open source tools to create a testing framework?” Simply put, it’s because you don’t need to reinvent the wheel! You may think you’re the only person who has encountered a particular testing problem, but I assure you that isn’t true. Many other people have likely experienced the same issue and have created and shared open source tools and resources in the community that can solve your challenge.

So how do we choose open source tools? First, you’ll be guided by the programming language you’re using. Also, check GitHub to see how many stars and forks, as well as how active the project is. Are people replying and does the community appear active? Is there solid documentation? Often when a project fails, it’s because it lacks good documentation. Conversely, good documentation can really help you get started easily.

Also, choosing an open source tool should really be an effort of the entire team. Everyone should be involved putting in their own concepts and ideas. First do you your research, and then do a proof of concept. Try out an open source tool with your team to determine if it’s a good fit for you.

Step 4. How to Test the Right Things

 

We know automation is important, so now we need to determine which tests to automate. But if automation is so valuable, shouldn’t we automate everything? As great as that sounds, the answer is a definite no. The fact is that automation takes time. It takes time to implement and it takes time to maintain, so we have to think critically about what to automate.

I believe you should prioritize automating tests that have high value for you, the team, and your organization as a whole. For example, if you’re testing an online shopping site, the checkout process may hold the highest value for your organization simply because of the revenue potential.

How do you identify what you do want to test?

The first step is to identify the main application flows that must always work. Ask yourself a few key questions:

• How bad is it if this feature/behaviour breaks?
• How much value does the test have?
• How big is the risk that mitigates?

 

Using our example of an online shopping site, we might decide that the following application flows are the most critical:

 

• Users can login
• Users can register their accounts
• Product images display correctly
• Items can be added to the shopping cart
• Payment can be collected

 

It’s important to know that you don’t need to write hundreds of tests to have your website tested! As in the example above, we can start with five prioritized tests for now, and then use analytics and user traffic data to help you evaluate the answers and determine the browsers, versions, and operating systems to test.

Putting a test into practice

Ultimately, using our example, we could have 5 tests executed in different browser, version, and operating system combinations, which then would give us around 200 test executions.

To put this into practice, you might look at the data and determine that the majority of your site users are using Chrome, Firefox and Safari.

Then, you can look further to determine the browser versions that are used most often, and then do the same for OS and device.

Based on our research thus far, we have identified our key application flows that will give us the most value. These will give us security knowing that testing these lowers the risk of something going wrong. Also, by looking at the analytics, we’ve decided to test on:

• Three different browsers (Chrome, Firefox, Safari)
• Two different OS (Windows, OS X)
• Three different screen resolutions
• Seven different browser versions (Chrome 71-74, Firefox 64-65, Safari 12)

This gives us the assurance that we are automating the highest value tests on the platforms being used most often.

Now that we know WHAT to automate, let’s turn to a few best practices for making your automated tests successful. Here are five best practices.

• Focus on reusability and maintainability. Avoid code duplication across tests and helper classes/methods.
• Every test must be autonomous. Tests can run in any order without depending on each other!
• Write tests and code only for the current requirements. Avoid complex designs that consider potential future use cases.
• Get familiar with software design patterns. They can benefit automation testing as well.
• Base your work on testing plans and/or strategies, not on tools. Don’t choose a tool and then look for ways to use it. Start with your strategy and plan, and only then should you consider the tools that can help.

Step 5. Where—and when—to run tests

The answer to these questions will really depend on which stage of testing you’re in.

Waterfall

Waterfall is the legacy that we are hopefully moving away from. For example, if you’re in waterfall, you may be doing a lot of manual testing. In this scenario, everyone may be working together towards a major release, possibly doing joint bug-hunting sessions together to prepare. In this situation you’re likely doing local testing on desktops, and it’s common to have shared testing environments. Due to the nature of manual testing, you just won’t have time to test more often.

Fast Waterfall

If you’re in fast waterfall, OK—that’s a good start when moving from manual testing to automation. It allows for a safe environment to learn how to automate and do proof of concepts for different approaches. Fast waterfall is where automation begins. Here, we start enabling testing to happen more often, perhaps one or more times per day. Tests are triggered manually and often combined with manual testing that takes place during the pre-release phase. We need to learn how our tests are evolving and working properly. In this scenario, we may still be testing locally or you may have built your own grid, and you may begin to dabble in cloud-based testing.

Continuous Integration 

Once you move to Continuous Integration, automation dominates. With CI, all high-value tests are automated, everyone can execute them, they are part of the CI pipeline and give fast feedback. This is the true benefit of automation. Manual testing will still happen, but will be used more often for debugging or exploratory testing. A cloud infrastructure is vital for success at this stage, because maintaining your own test grid is difficult and expensive—and you need comprehensive browser and operating system coverage.

Continuous Delivery 

With CD, the development process is fully automated. All teams should be able to do testing and development. Manual is used for exploratory testing only. Tests are also used to validate production deployments. When are you running tests? All the time: pre-commit, commit, after merge/pull requests, after merging, release, etc.

The result is that automated tests are fully resilient and trustworthy, and can be used in all environments. Again, at this stage, you should be trusting a cloud-based testing platform to ensure you can be confident in the tests across multiple browsers, devices and operating systems.

Conclusion

Now you’re ready to jump-start your test automation! If you want to learn even more about test automation, check out our EuroSTAR Online programme.

——————————————————————————————–

Author: Diego Molina, Software Engineer at Sauce Labs

Diego is part of the live testing team at Sauce, helping users get the most out of their manual and exploratory testing. He is part of the Sauce Labs Open Source Program Office, helping other Saucers get started with open source. Diego has put in a great deal of time working on the Selenium project, especially in improving the documentation and expanding into different languages so people across cultures can use Selenium to simplify and enhance their continuous testing efforts.  Diego runs the Continuous Testing Meetup Berlin, a group that provides an environment for local testers to share knowledge and participate in the community.

Testers worldwide are faced with similar issues and challenges in their test automation journey. The most common issues teams face pertaining to test automation are that scripts are brittle, fail too often and require a lot of maintenance.

This mostly happens because we are scripting everything at the same level, and when something changes at business level or at a UI level or at the data level, we end up going to the same script every time and trying to alter it.

So, what does it take to be in sync and achieve maintainable test automation?

1. Less Work to be Done: We need to reduce time of creation and try to offload it to machine instead of manual scripting. This is where a robust tool with good recorder, object spy and authoring capabilities can help.
2. More Hands to Help: We need to focus on getting more stakeholders and people involved and participate in the process, by contributing in some way to the overall process of test automation.

We can make this happen by understanding the different layers of test automation and how to separate them.

Maintainable test automation: Let’s consider the case of a user transferring money to another person and see how these layers help.

Layer 1: Business Layer

• This layer expresses business intent.
• It is agnostic of the UI or the web application itself, testing tool and interaction code, for example: Create User, Approve User, Login User etc.
• This layer will change only if business logic itself changes. So, it survives across UI implementations (web or mobile or desktop), and survives architectural changes.
• In this layer we can talk in the language of the business, without assuming anything about the UI, tool or code.

In Sahi Pro, this layer is implemented as Scenario files. This is a csv file which can be edited in our Editor or via MS Excel. A sample scenario looks like this:

 

Notice how the business layer does not assume anything about the user interface. It does not matter if this is on a web application or on a mobile app.

Also, each of the steps in this business flow will correspond to functions or methods implemented separately.

Layer 2: Implementation Layer

This layer implements the business keywords specified in the previous layer

• Understands interactions between different actions performed on UI
• Library file with implementation of keywords used in Business Layer
• Will change if interaction flow changes

 

Note the name of the method corresponds to the relevant step in the scenario file, the number of arguments and their values are also picked from the provided scenario, while the actual code for the step is implemented inside the method.

 

Layer 3: Accessor Repository Layer

This is the layer where we store every element’s locator and accessors.

• This is the Central repository of all elements in the automation code
• This changes when a particular element changes in the application UI (for example, due to HTML/Javascript changes in web interfaces)

As you see now, the business layer will be where you will define the flows and actions for every use case. This is high level and mostly worded in natural language. So, eventually your business team as well as manual testers will be able to write the scenario files easily.

The implementation layer is where you take each action and define the methods containing steps to perform that action. This is easily doable using any automation tool. With Sahi Pro, you can use record and playback as well as write your own using simple java script. These functions are stored in library files. These will change only if the flow or sequence of steps changes. These functions are not hard coded with your object ids, so they will not change even if an object’s locator changes. The locators are all in the Accessor Repository layer.

The Object locators are saved in Accessor Repository (AR) files.

In Sahi Pro, you can create an AR automatically as you record your script, and you can also have a single AR for multiple scenarios and scripts. This AR is the place you need to go and edit the object locator if anything changes.

Conclusion

• Automation code is most useful when there is a lot of change planned in your application
  • Acts as a safety net and guideline
  • Automation code should not be thrown away when application technology changes
• Building the right layers and strictly following them helps in minimal maintenance efforts and long-lived useful automation scripts

 

Find out more about maintainable test automation at the EuroSTAR Conference, Nov 11-15. Book your tickets now!

——————————————————————————————–

Author: Nishi Grover Garg – Product Evangelist and Head of Trainings, Sahi Pro

Nishi is a corporate trainer, an agile enthusiast and a keen tester at heart! With 11+ years of industry experience in Agile environment, she has worked in various roles as a hands-on tester, automation developer, a freelance consulting trainer and currently works with Sahi Pro as the Evangelist and Trainings Head. Nishi is passionate about training, organizing testing community events and meetups, and has trained hundreds of professionals and teams on agile, test automation, QA bootcamps and DevOps courses.

Nishi is also a writer on technical topics of interest in the industry and has numerous articles published at numerous popular forums and her own blog testwithnishi.com where she writes about the latest topics in Agile and Testing domains. Connect with her at @testwithnishi