Thanks to JetBrains for providing us with this blog post.
This post was originally published on the JetBrains Qodana Blog.
Competency in the TestOps field is now just as much an essential requirement for QA engineers as the ability to write automated tests. This is because of the ongoing development of CI/CD tools and the increasing number of QA engineers who work with pipelines (or the sequence of stages in the CI/CD pipeline) and implement their own.
So why is CI/CD such an excellent tool for quality control? Let’s find out!
Running Tests Automatically
Automated tests haven’t been run locally in what feels like ages. These days, CI/CD pipelines run tests automatically as one of their primary functions.
Pipeline configuration can be assigned to DevOps. But then we will be a long way from making use of the CI/CD tool’s second function: quality control, or more precisely, “quality gates”.
Quality Control Using Quality Gates
But what are quality gates? Let’s say the product code is like a castle. Every day, developers write new code – which could weaken the foundations of our castle or even poke holes in it if we are really unlucky. The purpose of a QA engineer is to test each feature and reduce the likelihood of bugs finding their way into product code. Lack of automation in the QA process could cause QA engineers to lose sleep, since there is nobody to watch over all the various metrics – especially at dangerous times, like Friday evenings when everyone wants to leave work and is hurrying to finish everything. An ill-fated merge at that moment can cause plenty of unwanted problems down the line.
This problem can be solved by building-in quality checks.
Each check deals with a different important metric. If the code doesn’t pass a check, the gates close, and the feature is not allowed to enter. A feature will only be merged into the product when it has passed all the checks and potential bugs have been fixed.
What Quality Checks can be Included in the CI/CD Pipeline?
We need to put together a list of checks to ensure that the process is as automated as possible. They can be sequenced in a “fail first” order. A feature must pass all the checks to get through the pipeline successfully. The initial checks ensure the app is capable of working: build, code style check, and static analysis.
“Build” speaks for itself: if the app fails to build, the feature does not progress. It is important to incorporate a code style check into your CI/CD pipeline to ensure the code meets unified requirements, as doing so allows you to avoid wasting time on this kind of bug during code reviews.
Static analysis is an essential tool for judging code quality. It can point out a vast number of critical errors that lead to bugs and decrease the number of routine and repetitive tasks for the QA team. Afterwards, developers should fix the detected issues and hand the code over for the testing stage.
We then continue with stage-two checks: unit tests with coverage analysis and coverage quality control, as well as integration and systems tests. Next, we review detailed reports of the results to make sure nothing was missed. At this stage, we may also perform a range of non-functional tests to check performance, convenience, security, and screenshot tests.
When developing a pipeline, we need to pay attention to 2 competing requirements:
- The pipeline must guarantee the best possible feature quality in light of your needs.
- Time spent running the pipeline should not slow down your workflow. It should generally take no more than 20 minutes.
Examples of Tools to Incorporate in Quality Checks
Code Style Highlighting
A code style is a set of rules that should be followed in every line of code in a project, from alignment rules to rules like “never use global variables”.
You might be wondering what style has to do with testers. The answer is a lot. A style check provides several benefits for QA experts, not to mention the rest of the team:
- A unified style helps developers work with the code and gives them more time to implement new features and fix bugs.
- A unified style allows you to dispense with manual code checks and use a CI/CD tool to run the checks instead.
Large companies usually have their own style guides that can be used as examples. For instance, Airbnb has a JavaScript style guide, and Google maintains several guides. You can even write your own, should you wish.
The choice of tools for code checking depends on the language. You can find a suitable tool on GitHub or find out which tools other teams use. Linters use bodies of rules and highlight code that fails to abide by them. Some examples include ktlint for Kotlin or checkstyle for Java.
Static Code Analysis
Static code analysis is a method of debugging by examining source code without executing a program. There are many different static code analyzers on the market.
We’ll now look at a platform we’re developing ourselves – Qodana. The significant advantage of this code analyzer is that it includes a number of inspections that are available in JetBrains development environments when writing code.
Many of you probably use an IDE-driven approach, where the IDE helps you write code and points out bugs such as suboptimal code usage, NullPointerExceptions, and duplicates.
But unfortunately, you can never be sure all the critical problems found by the IDE were fixed before the commit. However, you can ensure that the issues will be addressed by incorporating Qodana into your CI/CD pipeline.
Qodana, the latest addition to the family of products from JetBrains, is a cutting-edge static analysis platform designed to help developers and QA engineers improve their code quality, making it more efficient, maintainable, and bug-free. Its static analysis engine is the only solution on the market that brings native JetBrains IDE code inspections to any CI/CD pipeline. The platform provides an overview of project quality and lets you set quality targets, track progress, and automate routine tasks like code reviews.
If you can’t fix everything at once, you can select critical problems, add them to the baseline, and gradually work your way through the technical debt. This allows you to avoid slowing down the development process while keeping the problems that have been found under control.
Test Coverage
Test coverage is a metric that helps you understand how well your code has been covered by your tests (generally unit tests).
Here, you need to define the minimum coverage percentage you want to support. The code won’t be able to go live until it has been covered sufficiently by the tests. The minimum percentage is established empirically, but you should remember that even 100% coverage may not completely save your code from bugs. According to this article from Atlassian, 80% is a good figure to aim for.
Different coverage analyzers are available for other languages, such as Jacoco for Java, Istanbul for JavaScript, or Coverage.py for Python. You can build all these analyzers into your CI/CD pipeline and track the metrics with ease.
Shaping the Release Process
In addition to automatically running tests and ensuring particular code quality requirements are satisfied, the CI/CD tool lets testers organize the release process.
The release process can be complex and depend on many different manual actions. It is often a completely manual process: the artifact is created by a developer, then passed to the testers for checks, and finally comes to the person who knows how to roll it out for the go-live. Once again, there are a lot of potential choke points here. For instance, one of those people could fall ill or go on vacation.
An effective release process will look different for each team, but it will generally include the following steps:
- Each change in the Git branch triggers a build of the app.
- The build undergoes quality checks and does not become part of the main branch until it passes all the checks successfully.
- A release candidate is taken from the release branch or the main branch: this fixes the version and guarantees that nothing will go live unless it has been tested and has not been changed afterwards. This helps with tracking releases and all the changes they include. In addition, storing artifacts of the stable version makes it possible to revert to them quickly in the event of an unsuccessful release.
- The release candidate is tested and undergoes final checks.
- The release candidate goes live. This may be either a manual or automated pipeline launch, if the release candidate passed all the checks at the preceding stage. The choice between an automatic release process and a manual one will depend on how frequent and important the releases are, as well as the preferences among team members and the convenience of the rollout.
Any CI/CD system allows you to set up this type of process, which should be convenient for the whole team, including the testing team.
Given the factors outlined above, we believe following these basic rules will help ensure an easy and efficient release process:
- Artifacts must be ready for download and testing, ideally stored in one place.
- As many checks and tests as possible must be automated.
- All complex operations with builds should be as automated as possible.
- All builds that will go live should be recorded and remain available for a certain period after release. This will help if you need to investigate errors in the production version, reproduce bugs, or just track the history.
We would also like to remind you that if quality metrics are not controlled automatically and are not actionable, they are useless, as there’s no way to guarantee that these metrics will be adhered to.
Implement pipelines, automate processes, and use static code analysis!
Your Qodana team
Author
Alexandra Psheborovskaya, QA Lead and Product Manager at JetBrains
JetBrains is a global software company that creates professional software development tools and advanced collaboration solutions trusted by more than 12.8 million users from 220 counties and territories. Since 2000, JetBrains has built a catalog of 34 products, including PyCharm, IntelliJ IDEA, ReSharper, PhpStorm, WebStorm, Rider, YouTrack, Kotlin, and Space, a new integrated team environment.
Qodana is the code quality platform from JetBrains. It provides a project overview and lets developers and QA engineers set up quality gates, enforce project-wide and company-wide coding guidelines, better plan refactoring projects, and perform holistic license audits. Qodana’s static analysis engine enriches CI/CD pipelines with all of the smart features of JetBrains IDEs, supports 60+ languages and technologies, and allows analysis of unlimited lines of code.
JetBrains is an EXPO Platinum partner at EuroSTAR 2023, join us in Antwerp