Thanks to Karate Labs for providing us with this blog post.
A key point that many teams fail to ponder: “Is test-automation a first-class citizen of your development pipeline?”
This is the second part of a series of articles covering the finer aspects of test automation in detail. Read the first part here: The Test Automation Capability Map.
What happens when Developer Experience is not prioritized? Does this situation (see picture below) look familiar where you use a separate tool and workflow for authoring test-automation artifacts?
Let us zoom in on three of the Test-Automation capabilities which fall under the category of Developer Experience.
IDE Support
The development team spends the most time within their IDE of choice (e.g., IntelliJ, Visual Studio Code etc.). If test-automation requires a completely different tool, user-interface, and workflow, this has several implications for your team:
- Switching between tools takes developers out of their “flow state.”
- Developers are less likely to run tests before checking-in or merging code. This leads to an inefficient feedback loop, where failures are detected only when tests are run later.
- Developers are less likely to contribute and maintain tests. This results in dysfunctional teams where developers “chuck things over the fence” to the “QA team”. It is very common to find developers and QA teams operating in silos where bugs that “escape” to production result in finger-pointing and blame-games.
Self-Hosted or On-Prem
An aspect often overlooked is whether any sensitive data is leaving your safe-zone and coming to rest beyond your organization’s security perimeter or firewall. If you use a SaaS tool that is not self-hosted, this typically is the case.
Even though integration and end-to-end testing environments should ideally use fake or synthetic data, there will be cases where “production-like” data will be needed to simulate real business-scenarios. Many teams extract a “cut” of production data for a staging environment with some data-masking or sanitizing applied. Equally or more important than test-data is test-configuration, such as database-secrets, authentication-tokens, and passwords.
Given that a lot of teams operate in the public-cloud even for pre-production environments (e.g., AWS, Azure and GCP) it is even more important that the greatest care is taken to protect not just your test-data, but the configuration and locations and URLs of test-servers and other infrastructure. So, the critical question you should ask – Are my tests being stored in somebody else’s cloud?
Version Control, History, and Diffs
Self-hosted or not, the situation worsens if your test-automation artifacts are being managed in a separate tool or repository. The implications of having tests in a separate tool and workflow are understated, but significant
- You will lose the ability to eyeball the changes in your tests, side-by-side with the corresponding changes to code.
- There is less pressure (or no “forcing function”) to add or edit tests when code-changes are made. This discipline can make the difference between a high-performing team or one that lacks confidence to ship as often as needed.
- Tests are the best documentation of “what the system does.” When you cannot see the history of changes to tests side-by-side with the code commit history, you lose a valuable chunk of documentation, and you are left with an incomplete picture of how the software evolved.
- Your Continuous Integration job becomes more complex at the point where tests must be run. Instead of getting code and tests in one atomic “checkout” or “git clone” operation, you are forced to perform an extra step to download the tests from somewhere. Keep in mind that you also need to ensure that the version of the tests corresponds to what is being tested.
In addition, if you are using a tool with a “no-code” UI to author tests, it is quite likely that you lose the basic ability to see diffs and history even just for your tests. Some tool vendors have ended up having to build version-control into their user-experience, re-inventing what comes naturally to teams that use Git to collaborate.
Some things are best expressed as code. At the end of the day, everyone agrees that code-diffs are a superior Developer Experience.
To summarize, for your test-automation to be a first-class citizen of your development pipeline you need to use a tool that integrates into the team’s IDE of choice and stays close to the code being tested. The result of that is shown below.
An interesting observation is that: this is exactly the developer experience you expect for unit-tests!
So which kind of team do you want to be in?
Happy Testing!
Author
Peter Thomas, Co-founder & CTO, Karate Labs
Peter is recognized as one of the world’s top experts in test automation. He brings 25 years of industry experience from which he has been in open source for the last 18 years. He has worked at Yahoo and Intuit. As part of the API platform leadership at Intuit, Peter created “Karate” the open-source solution unifying API, UI & Performance testing. Peter was one of only 15 chosen by GitHub for a grant in India 2021. He co-founded Karate Labs Inc in Nov’21 to accelerate the adoption of Karate with the mission of making test automation fun and collaborative. Karate Labs is a Y Combinator backed company.
Karate Labs is a Platinum Partner at EuroSTAR 2023. Join us at Antwerp Zoo June 13-16, in a 4 day celebration of testing. Learn from 68 expert speakers and connect with your peers at Europe’s Best Testing Event. Book your tickets by Jan 31 and save 15% or book your team and save up to 40%.