EuroSTAR Conference

Europe's Best Software Testing Conference

Test Automation

The A-Z of Mobile Test Automation

by

Thanks to ACCELQ for providing us with this blog post.

Did you know? There are 5.48 billion unique mobile phone users across the globe. That’s almost 68.6% of the world’s population. No doubt, mobile app testing is soaring in popularity.

Mobile App Testing Is the Need of the Hour


About 59.72% of web traffic today comes from mobile phones. Imagine the quality and performance levels today’s teams must meet to cater to this web traffic. A top priority for teams is ensuring each mobile user has a seamless, secure, and satisfying experience every time they pick up the phone.

Crafting and maintaining this top-notch user experience demands rigorous and continuous testing. The right approach to mobile app testing can bring several benefits to the table. It can:

  • Improve user experience and boost retention rates
  • Reduce the frequency and complexity of bugs
  • Bring much-needed stability into mobile apps
  • Test if new features, changes, and enhancements are working properly
  • Boost user ratings and downloads and improve public perception

But Traditional Approaches to Mobile App Testing Are No Longer Scalable

Device fragmentation is at an all-time high. Testing the rich diversity of browsers, devices, and platform versions would require hiring an army of testers. Add to it the unique characteristics of different types of educational, lifestyle, social media, productivity, and gaming apps and the way they are built. For example:

  • Native apps are developed specifically for a single mobile platform such as iOS or Android. Built using the platform’s native programming language and development tool, they are typically faster and more responsive than other types of apps. Since they take advantage of device-specific features such as sensors and Bluetooth, they provide a more interactive UI/UX and have lesser compatibility issues.
  • Web apps are generally developed for mobile devices and accessed via any internet browser on a mobile device. They do not need any storage space and are constantly updated. They dramatically reduce the business costs associated with development and maintenance. They are responsive websites that adapt the user interface to the user’s device.
  • Hybrid apps are developed using web technologies such as HTML and CSS and wrapped in a native container that allows them to run on multiple platforms. Typically, they are easier and faster to develop than native apps but may not be as fast or responsive. However, hybrid apps are ideal in situations where high performance and full device access are not key requirements.

Mobile app testing isn’t just about writing a handful of test cases and running a few tests. Several challenges make testing a nightmare.

Setup

As depicted below, you must first feed your test script to a Selenium tool. Selenium then sends automation commands to an Appium Server, which logs results in a console. The Server also invokes vendor-specific configuration to execute commands via a simulator and an emulator.

Functional

When it comes to functional testing, several aspects of the mobile app have to be continuously and carefully tested — from web views to mouse actions, maps to images.

Miscellaneous

Mobile apps must also be continuously tested for interruptions. Evaluating how the app behaves during low battery, when a notification pops up, or when a call is received is equally important. In addition, it pays to test:

  • User experience during installation/uninstallation
  • UI with all forms of gesture navigation
  • When users don’t have permissions
  • When the network is not available
  • If your app updates correctly from previous versions

Building and maintaining a QA lab is extremely difficult, with varying mobile architectures, app security requirements, and feature demands. Since every mobile app must be tested for functionality, compatibility, usability, performance, security, and localization, it is critical to automate the testing process. Taking the mobile test automation route can open doors to several time, cost, and efficiency benefits.

Investing in the Right Test Automation Framework Is Key

A modern test automation framework can simultaneously test apps on multiple platforms and quicken the feedback cycle. It can maximize test coverage, enable 24/7 text execution, and allow reusability of test cases. It can also pave the way for seamless scalability – regardless of how big the app grows or how large the user base becomes.

But with so many frameworks available in the market today, how do you make the right choice?

  • How do you ensure the framework keeps up with the pace of app changes?
  • How do you efficiently simulate real-world user scenarios?
  • How do you ensure integration with backend systems?
  • How do you handle different types of apps and different types of mobile platforms?
  • How do you ensure continuous testing?

How To Choose the Right Mobile Test Automation Framework

If you want to transform your testing results, here are some things to keep in mind:

  • Identify your testing requirements
  • Evaluate different solutions
  • Consider ease of use and the overall learning curve
  • Look for a solution that is easy to maintain and scale and is extensible
  • Look for a solution that supports different app types and mobile platforms
  • Invest in a Unified Test Automation platform
  • Evaluate the tool’s DevOps – CI/CD integration capabilities
  • Assess reporting and analytics capabilities
  • Consider support features and evaluate community support
  • Take into consideration the cost and feasibility

How ACCELQ Handles Mobile Test Automation

As an intelligent, cloud-based mobile test automation platform, ACCELQ enables seamless multi-channel automation across the mobile tech stack. The tool takes a revolutionary approach to business assurance in a multi-packaged app environment. It allows for codeless mobile test automation that handles real-world complexities and presents a unified view of the quality lifecycle.

ACCELQ is a market leader in test automation and test management. Its automation flow recorder is coupled with a powerful Natural Language no-code editor. It can execute test automation across different mobile OSs and is agnostic of development frameworks. In addition, the tool’s design-first approach with inbuilt modularity means there is no need for custom frameworks.

  • Codeless: ACCELQ’s codeless capabilities allow teams to write and run test cases with zero setup and no coding. It enables testers to automate without the need for programming skills.
  • AI-powered: Being a true no-code mobile test automation platform, ACCELQ’s AI-powered capabilities automate and execute tests across different OSs and devices. The tool’s advanced mobile object handling capabilities eliminate test flakiness.
  • Lifecycle automation: ACCELQ allows for mobile test automation across the lifecycle of mobile apps. Using ACCELQ, teams can easily set up, design, develop, execute, and track mobile test automation.
  • Unified flow: ACCELQ offers full-blown version control, branching, and merging capabilities, all in one unified collaborative cloud platform. Teams can use ACCELQ to enable mobile, web, API, backend, and full-stack automation in the same unified flow.
  • Cross-device: ACCELQ’s Integrated Device Cloud Labs allow for seamless cross-device testing using a simple Plug & Play model.
  • High coverage: ACCELQ’s app universe and analytic-based algorithms drive automated test planning, ensuring coverage.
  • Actionable reports: ACCELQ offers dynamic live results views with actionable reports to trigger reruns. Email notifications that fit into the process allow for quick and effective decision-making.
  • Seamless support: The tool offers seamless support across popular mobile dev frameworks, including Android, iOS, React, Ionic, and Apache Cordova.
  • Robust and sustainable: The ACCELQ platform is robust and sustainable and offers automation capabilities that are significantly low on maintenance.
  • Self-healing: The tool’s self-healing element identification drastically enhances the quality and reliability of tests.
  • Low maintenance: Referential integrity across test assets hugely reduces maintenance and upkeep.

The Way Forward

If you want to meet the expectations of the constantly growing mobile user base, it’s time to take mobile testing more seriously. Embark on the mobile test automation journey today to make your tests more reliable and predictable and your apps more functional and secure.

Author

Nishan Joseph, VP-Sales Engineering,
ACCELQ

Nishan is a highly accomplished and dynamic leader who has been working in the technology space for over a decade. He is known for his ability to build strong partnerships with long-term strategic goals. Nishan leads the Sales Engineering division, while also overseeing some of the larger global Strategic Accounts for the company.

ACCELQ is an EXPO Exhibitor at EuroSTAR 2023, join us in Antwerp.

Why Do Testers Need CI/CD Systems?

by

Thanks to JetBrains for providing us with this blog post.

This post was originally published on the JetBrains Qodana Blog.

Competency in the TestOps field is now just as much an essential requirement for QA engineers as the ability to write automated tests. This is because of the ongoing development of CI/CD tools and the increasing number of QA engineers who work with pipelines (or the sequence of stages in the CI/CD pipeline) and implement their own.

So why is CI/CD such an excellent tool for quality control? Let’s find out!

Running Tests Automatically

Automated tests haven’t been run locally in what feels like ages. These days, CI/CD pipelines run tests automatically as one of their primary functions.

Pipeline configuration can be assigned to DevOps. But then we will be a long way from making use of the CI/CD tool’s second function: quality control, or more precisely, “quality gates”.

Quality Control Using Quality Gates

But what are quality gates? Let’s say the product code is like a castle. Every day, developers write new code – which could weaken the foundations of our castle or even poke holes in it if we are really unlucky. The purpose of a QA engineer is to test each feature and reduce the likelihood of bugs finding their way into product code. Lack of automation in the QA process could cause QA engineers to lose sleep, since there is nobody to watch over all the various metrics – especially at dangerous times, like Friday evenings when everyone wants to leave work and is hurrying to finish everything. An ill-fated merge at that moment can cause plenty of unwanted problems down the line.

This problem can be solved by building-in quality checks.

Each check deals with a different important metric. If the code doesn’t pass a check, the gates close, and the feature is not allowed to enter. A feature will only be merged into the product when it has passed all the checks and potential bugs have been fixed.

What Quality Checks can be Included in the CI/CD Pipeline?

We need to put together a list of checks to ensure that the process is as automated as possible. They can be sequenced in a “fail first” order. A feature must pass all the checks to get through the pipeline successfully. The initial checks ensure the app is capable of working: build, code style check, and static analysis.

“Build” speaks for itself: if the app fails to build, the feature does not progress. It is important to incorporate a code style check into your CI/CD pipeline to ensure the code meets unified requirements, as doing so allows you to avoid wasting time on this kind of bug during code reviews.

Static analysis is an essential tool for judging code quality. It can point out a vast number of critical errors that lead to bugs and decrease the number of routine and repetitive tasks for the QA team. Afterwards, developers should fix the detected issues and hand the code over for the testing stage.

We then continue with stage-two checks: unit tests with coverage analysis and coverage quality control, as well as integration and systems tests. Next, we review detailed reports of the results to make sure nothing was missed. At this stage, we may also perform a range of non-functional tests to check performance, convenience, security, and screenshot tests.

When developing a pipeline, we need to pay attention to 2 competing requirements:

  1. The pipeline must guarantee the best possible feature quality in light of your needs.
  2. Time spent running the pipeline should not slow down your workflow. It should generally take no more than 20 minutes.

Examples of Tools to Incorporate in Quality Checks

Code Style Highlighting

A code style is a set of rules that should be followed in every line of code in a project, from alignment rules to rules like “never use global variables”.

You might be wondering what style has to do with testers. The answer is a lot. A style check provides several benefits for QA experts, not to mention the rest of the team:

  1. A unified style helps developers work with the code and gives them more time to implement new features and fix bugs.
  2. A unified style allows you to dispense with manual code checks and use a CI/CD tool to run the checks instead.

Large companies usually have their own style guides that can be used as examples. For instance, Airbnb has a JavaScript style guide, and Google maintains several guides. You can even write your own, should you wish.

The choice of tools for code checking depends on the language. You can find a suitable tool on GitHub or find out which tools other teams use. Linters use bodies of rules and highlight code that fails to abide by them. Some examples include ktlint for Kotlin or checkstyle for Java.

Static Code Analysis

Static code analysis is a method of debugging by examining source code without executing a program. There are many different static code analyzers on the market.

We’ll now look at a platform we’re developing ourselves – Qodana. The significant advantage of this code analyzer is that it includes a number of inspections that are available in JetBrains development environments when writing code.
Many of you probably use an IDE-driven approach, where the IDE helps you write code and points out bugs such as suboptimal code usage, NullPointerExceptions, and duplicates.

But unfortunately, you can never be sure all the critical problems found by the IDE were fixed before the commit. However, you can ensure that the issues will be addressed by incorporating Qodana into your CI/CD pipeline.

Qodana, the latest addition to the family of products from JetBrains, is a cutting-edge static analysis platform designed to help developers and QA engineers improve their code quality, making it more efficient, maintainable, and bug-free. Its static analysis engine is the only solution on the market that brings native JetBrains IDE code inspections to any CI/CD pipeline. The platform provides an overview of project quality and lets you set quality targets, track progress, and automate routine tasks like code reviews.

Interactive inspection report in the Qodana code quality platform.

If you can’t fix everything at once, you can select critical problems, add them to the baseline, and gradually work your way through the technical debt. This allows you to avoid slowing down the development process while keeping the problems that have been found under control.

The updated baseline in the Qodana code quality platform.

Test Coverage

Test coverage is a metric that helps you understand how well your code has been covered by your tests (generally unit tests).

Here, you need to define the minimum coverage percentage you want to support. The code won’t be able to go live until it has been covered sufficiently by the tests. The minimum percentage is established empirically, but you should remember that even 100% coverage may not completely save your code from bugs. According to this article from Atlassian, 80% is a good figure to aim for.

Different coverage analyzers are available for other languages, such as Jacoco for Java, Istanbul for JavaScript, or Coverage.py for Python. You can build all these analyzers into your CI/CD pipeline and track the metrics with ease.

Shaping the Release Process

In addition to automatically running tests and ensuring particular code quality requirements are satisfied, the CI/CD tool lets testers organize the release process.

The release process can be complex and depend on many different manual actions. It is often a completely manual process: the artifact is created by a developer, then passed to the testers for checks, and finally comes to the person who knows how to roll it out for the go-live. Once again, there are a lot of potential choke points here. For instance, one of those people could fall ill or go on vacation.

An effective release process will look different for each team, but it will generally include the following steps:

  1. Each change in the Git branch triggers a build of the app.
  2. The build undergoes quality checks and does not become part of the main branch until it passes all the checks successfully.
  3. A release candidate is taken from the release branch or the main branch: this fixes the version and guarantees that nothing will go live unless it has been tested and has not been changed afterwards. This helps with tracking releases and all the changes they include. In addition, storing artifacts of the stable version makes it possible to revert to them quickly in the event of an unsuccessful release.
  4. The release candidate is tested and undergoes final checks.
  5. The release candidate goes live. This may be either a manual or automated pipeline launch, if the release candidate passed all the checks at the preceding stage. The choice between an automatic release process and a manual one will depend on how frequent and important the releases are, as well as the preferences among team members and the convenience of the rollout.

Any CI/CD system allows you to set up this type of process, which should be convenient for the whole team, including the testing team.

Given the factors outlined above, we believe following these basic rules will help ensure an easy and efficient release process:

  • Artifacts must be ready for download and testing, ideally stored in one place.
  • As many checks and tests as possible must be automated.
  • All complex operations with builds should be as automated as possible.
  • All builds that will go live should be recorded and remain available for a certain period after release. This will help if you need to investigate errors in the production version, reproduce bugs, or just track the history.

We would also like to remind you that if quality metrics are not controlled automatically and are not actionable, they are useless, as there’s no way to guarantee that these metrics will be adhered to.

Implement pipelines, automate processes, and use static code analysis!

Your Qodana team

Author

Alexandra Psheborovskaya, QA Lead and Product Manager at JetBrains

JetBrains is a global software company that creates professional software development tools and advanced collaboration solutions trusted by more than 12.8 million users from 220 counties and territories. Since 2000, JetBrains has built a catalog of 34 products, including PyCharm, IntelliJ IDEA, ReSharper, PhpStorm, WebStorm, Rider, YouTrack, Kotlin, and Space, a new integrated team environment.

Qodana is the code quality platform from JetBrains. It provides a project overview and lets developers and QA engineers set up quality gates, enforce project-wide and company-wide coding guidelines, better plan refactoring projects, and perform holistic license audits. Qodana’s static analysis engine enriches CI/CD pipelines with all of the smart features of JetBrains IDEs, supports 60+ languages and technologies, and allows analysis of unlimited lines of code.

JetBrains is an EXPO Platinum partner at EuroSTAR 2023, join us in Antwerp

10 Points to Help You Choose the Right Test Automation Tool

by

Thanks to Testsigma for providing us with this blog post.

Making a decision to start test automation is easy but choosing a right test automation tool is not. There are teams that are spending a lot on hiring new manual testing resources but find it hard to invest in automation. The reasons could be many.

Sometimes, teams spend a lot of time exploring tools and get so overwhelmed with the information out there that they give up on the idea of automation altogether. Other times, choose a generic tool, start with automation but then, never get past the first few test cases.

In this article, we have put together some points that will help such teams navigate their search for the right test automation tool.

Points to Select the Right Test Automation Tool

1. Project Requirements:

There is no point in looking for a solution when you don’t know the problem. So, before you start exploring the various tools and technologies available in the market for test automation, you need to list down your project requirements and the problems you are looking to solve.

The list, in general, should answer the below questions:

  • Type of application that needs to be tested: It could be web, mobile, API or a desktop application.
  • Platforms that need to be tested: If yours is a desktop application, list down the operating systems that should be tested. If yours is a mobile application, then list down the supported mobile operating systems. If yours is a web application, then list down the supported browsers.
  • Language your application is built in: This can help if you are planning to use a programming language for automation.
  • Need for cross-browser testing/cross-device testing: If yours is a web application or a mobile application then you will, most probably, need this.

In addition, you could also add requirements that you deem important.

2. Team Skills / Learning Curve:

When selecting a tool for automation there could be 2 types of tools:

  • A codeless test automation tool
  • An automation tool that requires coding

If yours is a team that already has people that are skilled in some programming language then you can think of using an automation tool in that programming language. Or, if you plan to hire skilled people for automation then you don’t need to consider this point.

But, if you are planning to have an automation tool that will not need you to look for people with the required skillset, going for codeless automation tools will be a good idea. These tools allow the automation of test cases without the need for knowing a programming language.

Check this guide to know about Codeless Testing in detail.

3. Budget:

This is a very important aspect of choosing the correct automation tool. You might easily say that you will want a free tool because you don’t want to spend on automation if you can avoid it.

But, you also need to consider that the amount of time being spent on automation, the number of people working on the tool and the machines being used for automation also constitute the amount you spend on automation. So, consider below points before deciding the budget:

  • Cost of human resources being used for automation: If there is a tool that does not need you to hire new resources especially for automation, consider it a saving.
  • Time spent on learning the tool: If there is a tool that has a low learning curve, that is an indirect saving in the cost you might have spent in terms of the time the resources spend on learning the tool. Or hiring resources that are skilled in that particular tool.
  • Time being spent on automation: If there is a tool that makes it easy to create and maintain test cases, thereby saving time, consider it a saving in cost.
  • Cost of infrastructure: Talking about cloud and hosting, you can go for an ideal PHP hosting that gives an amazing managed hosting experience.

4. Ease of Test Case Creation and Maintenance:

Not every tool is made to handle all kinds of scenarios. So, to make sure that your chosen tool meets your needs, try automating a few test cases of your application to know if the tool suits your needs. That could be done with the trial version of a tool if your search has narrowed down to premium tools.

Also, to avoid spending more time in test case maintenance as compared to test case creation, make sure to choose a tool that fits your budget including the maintenance costs. There are tools that have the ability to self-heal the test cases in case of minor changes in the application.

Such tools help to reduce the cost of test case maintenance. Also, it helps if the tool supports pause and resume of test case execution for a better debugging experience.

5. Reusability:

To avoid writing the same code multiple times in multiple test cases and to avoid duplication of efforts, look for tools that allow the reuse of already created test steps in different test cases and projects.

6. Data-driven Testing:

If yours is an application that needs testing for a variety of data at multiple interfaces, it is important to choose a tool that supports data-driven testing.

7. Reporting:

Test case creation and test case execution would be useless if the reports were not useful so do go through all the features in the reporting supported by a tool. Few of them would be:

  • Screenshots for failed steps
  • Video for test execution
  • Stack trace for the error
  • A clear indication of failed test cases/steps
  • Time taken for execution of test steps and test cases is reported

8. Support for Collaboration:

If you are doing automation of a project for a client, the client will want to review the quality of automated test cases.

It will also be beneficial if other non-technical members of the team are able to automate/review the test cases. So, in such scenarios, look for tools that make collaboration with the management and clients easy.

9. Support for Tools for Integration:

If there are some process improvement or CI/CD tools that you already use or plan on using, make sure that you chose a tool that integrates with them.

10. Training and 24×7 Support:

Consider a scenario – you started using a tool for automation, and after automating about 10 test cases successfully, you got stuck on the 11th test case; you don’t know how to resolve the problem. You have looked at all possible forums but you don’t have a solution in sight. If you want to save the time, use a tool that has 24×7 support to resolve any problems you encounter.

Conclusion

Sometimes, teams decide to create their own test automation frameworks because they cannot find a right test automation tool that fit their testing requirements.

At the moment, there really are multiple types of test automation frameworks and tools available in the market that support automation on a varied variety of applications and are still being improved.

So, do go through the above points and spend some time looking at available test automation tools before thinking of implementing a framework on your own.

At the end, I would like you tell you about a test automation tool Testsigma. It is a cloud-based test automation tool that lets you automate test cases just in simple English – no coding required. In addition, you can automate your test cases for web as well as mobile apps at the same place. Do check it out if it meets your needs.

Frequently Asked Questions

How to Choose Automation Tool?
To choose the right test automation tool, you have to ensure it’s capable, powerful, flexible, and up to the mark for your project requirements. By saying capable, powerful, and flexible, your selected tool should be capable enough to manage all the test cases and test data smoothly. It should also be flexible enough to integrate with other third-party tools to extend and customize the functionality and make testing even easier.

Which Tool is Mostly Used for Automation Testing?
There is an abundance of testing tools available in the market that can be divided into two categories. The first one is the no code low code test automation tool which requires little or no knowledge of programming to perform any type of test automation.

What are the Criteria for Selecting a Test Automation Tool for Your Project?
Criteria to select a test automation tool are the following:

  • Capable- It should be able to manage the project test cases and test data efficiently.
  • Flexibility- It should be able to integrate with other third-party tools to extend the functionality tool.
  • Cost-effective- It should come into your project budget.
  • Learning curve- Learning to use the tool should not be challenging for your other team members.

Is Selenium the Best Testing Tool?
If you are from a developer background, then Selenium is a free and open-source tool project that provides various tools, resources, and libraries to make test automation easy for everyone.

When should We Choose Automation Testing?
When you required speed and accuracy at the same time, you should go for automation testing instead of manual testing, automation testing enables your team to do more in less time by providing the features like test cases to manage the project all test cases, test data management to manage the project test data, test labs to run the test case in the combinations of operating systems and browsers, etc.

Author

Shruti Sharma Testsigma writer and content marketer

Shruti Sharma

Shruti is a writer and a content marketer with more than 10 years of experience in testing and test automation, and has been associated with Testsigma since about 3 years. She loves to read, learn, and write in detail about testing, test automation and tools. In addition, she also writes fiction. One cause she deeply cares about is mental health and psychology.

Testsigma is an EXPO Gold partner at EuroSTAR 2023, join us in Antwerp


Is test automation a first-class citizen of your development pipeline?

by

Thanks to Karate Labs for providing us with this blog post.

A key point that many teams fail to ponder: “Is test-automation a first-class citizen of your development pipeline?”

This is the second part of a series of articles covering the finer aspects of test automation in detail. Read the first part here: The Test Automation Capability Map.

What happens when Developer Experience is not prioritized? Does this situation (see picture below) look familiar where you use a separate tool and workflow for authoring test-automation artifacts?


Let us zoom in on three of the Test-Automation capabilities which fall under the category of Developer Experience.

IDE Support

The development team spends the most time within their IDE of choice (e.g., IntelliJ, Visual Studio Code etc.). If test-automation requires a completely different tool, user-interface, and workflow, this has several implications for your team:

  • Switching between tools takes developers out of their “flow state.”
  • Developers are less likely to run tests before checking-in or merging code. This leads to an inefficient feedback loop, where failures are detected only when tests are run later.
  • Developers are less likely to contribute and maintain tests. This results in dysfunctional teams where developers “chuck things over the fence” to the “QA team”. It is very common to find developers and QA teams operating in silos where bugs that “escape” to production result in finger-pointing and blame-games.

Self-Hosted or On-Prem

An aspect often overlooked is whether any sensitive data is leaving your safe-zone and coming to rest beyond your organization’s security perimeter or firewall. If you use a SaaS tool that is not self-hosted, this typically is the case.

Even though integration and end-to-end testing environments should ideally use fake or synthetic data, there will be cases where “production-like” data will be needed to simulate real business-scenarios. Many teams extract a “cut” of production data for a staging environment with some data-masking or sanitizing applied. Equally or more important than test-data is test-configuration, such as database-secrets, authentication-tokens, and passwords.

Given that a lot of teams operate in the public-cloud even for pre-production environments (e.g., AWS, Azure and GCP) it is even more important that the greatest care is taken to protect not just your test-data, but the configuration and locations and URLs of test-servers and other infrastructure. So, the critical question you should ask – Are my tests being stored in somebody else’s cloud?

Version Control, History, and Diffs

Self-hosted or not, the situation worsens if your test-automation artifacts are being managed in a separate tool or repository. The implications of having tests in a separate tool and workflow are understated, but significant

  • You will lose the ability to eyeball the changes in your tests, side-by-side with the corresponding changes to code.
  • There is less pressure (or no “forcing function”) to add or edit tests when code-changes are made. This discipline can make the difference between a high-performing team or one that lacks confidence to ship as often as needed.
  • Tests are the best documentation of “what the system does.” When you cannot see the history of changes to tests side-by-side with the code commit history, you lose a valuable chunk of documentation, and you are left with an incomplete picture of how the software evolved.
  • Your Continuous Integration job becomes more complex at the point where tests must be run. Instead of getting code and tests in one atomic “checkout” or “git clone” operation, you are forced to perform an extra step to download the tests from somewhere. Keep in mind that you also need to ensure that the version of the tests corresponds to what is being tested.

In addition, if you are using a tool with a “no-code” UI to author tests, it is quite likely that you lose the basic ability to see diffs and history even just for your tests. Some tool vendors have ended up having to build version-control into their user-experience, re-inventing what comes naturally to teams that use Git to collaborate.

Some things are best expressed as code. At the end of the day, everyone agrees that code-diffs are a superior Developer Experience.

To summarize, for your test-automation to be a first-class citizen of your development pipeline you need to use a tool that integrates into the team’s IDE of choice and stays close to the code being tested. The result of that is shown below.

An interesting observation is that: this is exactly the developer experience you expect for unit-tests!


So which kind of team do you want to be in?
Happy Testing!

Author

Peter Thomas Co-founder & CTO Karate Labs

Peter Thomas, Co-founder & CTO, Karate Labs

Peter is recognized as one of the world’s top experts in test automation. He brings 25 years of industry experience from which he has been in open source for the last 18 years. He has worked at Yahoo and Intuit. As part of the API platform leadership at Intuit, Peter created “Karate” the open-source solution unifying API, UI & Performance testing. Peter was one of only 15 chosen by GitHub for a grant in India 2021. He co-founded Karate Labs Inc in Nov’21 to accelerate the adoption of Karate with the mission of making test automation fun and collaborative. Karate Labs is a Y Combinator backed company.

Karate Labs is a Platinum Partner at EuroSTAR 2023. Join us at Antwerp Zoo June 13-16, in a 4 day celebration of testing. Learn from 68 expert speakers and connect with your peers at Europe’s Best Testing Event. Book your tickets by Jan 31 and save 15% or book your team and save up to 40%.

The test automation capability map

by

Thanks to Karate Labs for providing us with this blog post.

Test Automation is serious business. With teams under pressure to ship more often for competitive advantage, ensuring the best-quality end-user experience is key.

The choice of a test-automation tool can make the difference between a team that has high velocity – or one that lacks confidence to release.

We have seen many teams that have made the mistake of writing an in-house test-automation framework. I have personally experienced the pain of developers who have inherited tests in such a framework multiple times.

Unfortunately, teams continue to write their own framework for many reasons. One is that development skills are valued more than domain-knowledge or knowledge about the end-user. Creating a test-framework “from scratch” is over-hyped, fueled by a multitude of influencers and bloggers.

A reason that few like to talk about is that it is easier to write some code instead of thinking through what it takes to actually test a given piece of software. You would have seen cases where a team spends months building a test-framework, only for a new team to come along and promptly decide that a re-write is required.

To help teams make the right choice, we have created this graphic to emphasize a few points. It can be used as a handy reference to compare test-automation tools and discuss the relative importance of the capabilities for your specific team.

Please pass this on to those who are responsible for driving standards and evaluating tools in your enterprise.

test automation capability map

To summarize:

  • Test Automation requires a large set of fundamental capabilities. Are you sure you can implement all these in-house, or would you be better-off using an industry standard solution?

  • Does the tool pay attention to Developer Experience? Keep in mind that tests must be maintained for as long as your systems are in production.

  • Does the tool bring developers and QA closer together or force them into different user-interfaces and workflows?

  • Can your tests be version-controlled just like your code? To be specific, can you see the history and the “diffs” between previous versions?

  • Is your test data within your firewall or going into “somebody else’s cloud”?

  • Is it easy to simulate end-user-flows and the core capabilities (your reason to exist) of your business?

  • Is there a mature community of users or someone that you can reach out to for support?

  • How useful is the documentation? Does it go into detail? Are there enough examples?

  • How easy is it for someone new to the tool to start using it productively?

The list of capabilities in the graphic is a result of many iterations and discussions. Do let us know if any test-aspect is missing, or if you have any other feedback!

And by the way, the boxes in blue are things that Karate does today.

Happy Testing!

Author

Peter Thomas Co-founder & CTO Karate Labs

Peter Thomas, Co-founder & CTO, Karate Labs

Peter is recognized as one of the world’s top experts in test automation. He brings 25 years of industry experience from which he has been in open source for the last 18 years. He has worked at Yahoo and Intuit. As part of the API platform leadership at Intuit, Peter created “Karate” the open-source solution unifying API, UI & Performance testing. Peter was one of only 15 chosen by GitHub for a grant in India 2021. He co-founded Karate Labs Inc in Nov’21 to accelerate the adoption of Karate with the mission of making test automation fun and collaborative. Karate Labs is a Y Combinator backed company.

Karate Labs is a Platinum Partner at EuroSTAR 2023. Join us at Antwerp Zoo June 13-16 in a 4 day celebration of testing. Learn from 65 expert speakers and connect with your peers at Europe’s Best Testing Event. Book your tickets by Dec 31 and save 20% or book your team and save up to 45%.

7 Automation Sessions to Improve your Test Automation Skills

by

The EuroSTAR Huddle Deep Dive is just around the corner. Taking place from 1-4 November, this live event will take a deep dive into all aspects of test automation. If you are an automation engineer and want to improve your automation skills, the Deep Dive week is the event you should be attending. Let’s have a look at what is happening.

Big Data and Flaky Tests

Adam Sandman & Denis Markovtsev are at the cutting edge of test automation. They plan to showcase their novel use of Big data in test automation. This event will explain how big data was utilised by Adam and Denis to examine and reduce test flakiness. Part of a research project, this talk will showcase their approach of analysing 500 websites by downloading their DOM trees and performing data analysis to see how best practices developed in theory will work in practices with these sites. 

What is Cypress?

What is Cypress? That is a question many of you might have some answer to. Cypress is becoming a popular tool in automation but what does it do? Marie Drake will discus Cypress as a tool is and look at the importance of visual testing and how to integrate visual testing plugins to Cypress.

Making use of Low Code Automation

Paul Grossman has many years of experience in automation. He is a fan of low code automation and so makes a great person to discuss the uses of low code automation. If you have thought about applying low code tools this talk will fill you in on its uses. Paul will explain how you can utilise low code automation and showcase its use with some demo’s too.

Learning From Mistakes in Automation

What are the common mistakes made in test automation? Corina Pip is here to describe and share her advice on the common mistakes in test automation from her experience in the field. This will be a great talk to learn the common pitfalls in automation that we end up in. The Dos and Don’ts of Automation will offer some real world experiences of when automation turns out to be not as useful as you might have hoped!

API Testing

API have become a bigger part of our everyday life. This means a bigger requirement for testing API’s. Julia Pottinger will discuss API automation and in particular Scenarios to consider when doing API Automation. Learn about the world of API’s where to get started, and how to start the process of testing API’s.

Ask Me Anything

Dorothy Graham has over 30 years in the software testing industry much of that focusing on automation. In this live event she will reflect of the week of live events, where automation is going but more importantly will answer your questions on test automation.

Testing the Tests

The week concludes with a well renowned test automation expert: Bas Dijkstra. Bas will ask and answer the question: Who Tests the Tests? In this lively session he will share how you can make quality control for your automated tests part of your testing and development process. He will also introduce the technique of mutation testing and how you might use it for checking the quality of your automated tests.

So seven talks that can only excite you about the new skills and takeaways that you could learn from these sessions. Remember to sign up here.

COVID-19: Automation as a Solution

by

The pandemic we lived in the past year has not only accelerated the transition to automated processes but is also leveraging investments that drive change and protect organizations in scenarios like the current one.

All sectors are feeling the effects of the pandemic today. In recent weeks, the discussion on the economic and productive impact of companies has deepened, which will naturally be felt in the present and future. One thing we can be sure of, the pandemic will bring profound changes in the corporate fabric, not only negative but opportunities to improve methodologies and processes. Some have already given COVID-19 responsibility for speeding up the digital transformation process, at least in organizations with the process under development or at an early stage. With employees at home, many companies have drastically reduced their productivity, this being a moment of reflection. It is in this reflection that the opportunities and the vision to succeed in the future arise.

Automation is one of the first areas to take a step forward in companies’ evolution. The economic and productivity challenges we face today can be partly solved by automating routines, processes, and actions.

Today, and even more in the future, organizations need cost-effective and robust processes on the path to sustainable digital transformation. Human resources are often assigned to tasks and routine activities. The diversity of interfaces and different degrees of complexities leads to an increased probability of error in key organizational processes. Using robotic software to perform these routine tasks, rather than humans, makes it possible to execute the processes remotely, in turn ensuring business continuity.

Automation should not be seen as a threat. It should be seen by decision-makers as a tool that ensures productivity, not only by automatism itself and by decreasing the probability of errors, but because it allows allocating human resources to other tasks that may be more priority for the organization.

At Noesis, as a result of our industry and the automation of our processes, today we are 100% operational, ensuring a quick and efficient response to our customers. We believe that at least up to 30% of organizations’ tasks can be automated through Robotic Process Automation (RPA). We are talking about multiple activities, from generating an automatic response to an email to implementing thousands of bots, each programmed to complete a specific task, based on the state of the art technologies based on Machine Learning and Artificial Intelligence.

As a basis for the automation of RPA processes, Noesis uses its background with more than 10 years of experience in the field of test automation, being at this moment of Lockdown an excellent weapon to combat the errors of the applications that support, not only our day-to-day work but the applications that support the economy, allowing the uninterrupted provision of online services.

For this reason, and because we need the services and applications on the web to remain available, Noesis also actively participates in detecting bottlenecks of availability through the execution of load tests and safety tests, in a preventive way. In this way, we have received more and more orders in these areas, with the primary objective of reducing our customers’ costs, by detecting errors much earlier in the software development cycle, be they functional, availability, or security errors.

The future is a little more uncertain today, but the challenges are the same and we will continue to be ready to find solutions.

Noesis is exhibiting at EuroSTAR 2021 Online. Check out the EuroSTAR 2021 programme, and join us September 28-30th.

Author

Tiago Honorato, DevOps & Automation Senior Manager at Noesis

Tiago has 10+ years of experience in SW Development and Frameworks creation and design. He is currently Senior Manager at Noesis in the automation area, and is responsible for managing test automation projects for Noesis clients. He is creator, and responsible for, an innovative solution that became a Noesis product.

First Steps to Approach Test Automation

by

Test automation is a top priority for today’s companies. All, or almost, are betting on this aspect to reduce the effort of carrying out manual tests, avoiding the turnover of people in the projects, motivating employees, and saving time in the commercialization of products and services. But before jumping into the automation topic, several key aspects must be considered to fulfill the goal we establish.

The first thing to do is to analyze those three important areas need to be included in any test automation process: tools, test environment, and scope; although it is not necessary to do it in this order, because they are related.

Tools

The first step is to consider which tool meets our needs. The important thing is to analyze our needs and observe the types of validation, architecture, and tools that allow us to build, manage and report the automation project. The process of selecting an automation tool for web or mobile is not the same if we are thinking of an SAP system. Therefore, in this case, it is necessary to take a more specific approach concerning technical support and tools that suit the type of system chosen.

Test Environments

We need software, licenses, and infrastructure, in addition to a log of executions and parameterization. Therefore, it is necessary to define the test data management strategy and the requirements for continuous testing. In many cases, it is important to keep in mind that to get these sandboxes up and running, we need a lot of data. This is an aspect that cannot be ignored.

Scope

In terms of reach, what we must evaluate is what we want to automate. The initial assessment should aim to define which business processes we are going to automate. To do this, we must identify what kind of applications we have, the repetitive tasks and what we want to automate. Likewise, we have to define at what level we will implement test automation.

With these three fundamentals well established, we can observe some additional steps, such as the identification of the pilot application, the definition of small validation steps, which must be carried out step by step to manage expectations properly, and the automation criteria, which are used to identify the stability of the application in a test state. In this sense, we have to be sure of what will be used and evaluate the application’s life cycles; if it will be used for a long time, if it is obsolete or if the test time is acceptable.

Return On Investment

Last but not least, we must look at ROI (Return On Investment), taking into account both automation costs, with the remaining costs that we will have. The important thing here is to estimate the break-even point for the development, support/maintenance effort, and the tasks that must be performed manually. We will find the balance when we reach approximately 80% profit (considering the use of automation) and 20% effort (that would be the total amount to automate 100% of the reach).

In this sense, each organization must define the levels of automation that it will implement, and its level of scales, starting from 0 that corresponds to having no automation, and gradually going through levels 1 to 4. The first level corresponds to a basic level; the second at a more productive level, where we already have a roadmap, defined objectives, and KPIs; the third with a more consolidated automation framework, in which we already work on processes and not just on applications; and a fourth, and last, more advanced, in which we have already introduced artificial intelligence. These levels give a priority reference to guide us and the ideal is to have a detailed scale of priorities. Thus, as we increase coverage by 60-80%, investment starts to decrease. This is an important aspect to consider before starting.

Ultimately, it is important to focus on the future uses of test automation, based on good market practices, as well as how to get the ROI we want to achieve and good ways to measure it.

Noesis are exhibiting at EuroSTAR 2021 Online. Looking to learn more on test automation? Check out the EuroSTAR 2021 programme, and join us September 28-30th.

Author

Eduardo Amaral, Quality Management Director at Noesis

Quality Management Director at Noesis – Portuguese QM Services Market Leader, Eduardo worked as analyst/developer, project manager, program manager and service delivery manager. He has a background and certifications in several areas, including quality assurance, project/program management, process management and product management. Over the last 21+ years, Eduardo has been involved in a variety of software project development lifecycles and solutions integration, at high levels of complexity and customer experience exposure, performed in some of the major national and international companies, from different business areas – Industry, Banking, Insurance, Telco, Retail, etc.