• Skip to main content

EuroSTAR Conference

Europe's Best Software Testing Conference

  • Programme
    • 2024 Programme Committee
    • 2023 Programme
    • 2023 Speakers
  • Attend
    • Venue & Hotels
    • Bring your Team
    • Testimonials
    • Why Attend
    • EuroSTAR Awards
    • Academic Discount
  • Partner
    • Download our EXPO Brochure
    • Partner Opportunities
    • Partner Testimonials
  • About
    • About EuroSTAR
    • FAQ
    • EuroSTAR Timeline
    • EuroSTAR Blog
    • Supporting Organisations
    • Contact Us
  • Book Now

DevOps

World Quality Report 2022-2023: Orchestrating Quality in Agile Organizations

May 10, 2023 by Lauren Payne

Thanks to Sogeti for providing us with this blog post.

With the ongoing evolution of Agile and DevOps addressing the need to release more and release faster, quality assurance plays a vital role at every stage of the release cycle.

In the 14th Edition of the World Quality Report, we see quality in the agile and DevOps environment viewed like an orchestra. Every element of the software development process comes together in harmony to complete the finished piece, like a perfect musical performance where quality is assured.

This is referred to as quality orchestration whereby the teams, skillsets, provisioning, automation practices, service virtualization, and more are viewed – and managed – as a seamless end-to-end whole. Collaboration in this orchestrated agile environment is key. It’s how test and quality engineers ensure their activities deliver the two most important quality objectives for the agile enterprise – excellent customer experience and business outcomes.

Our latest World Quality Report survey found that agile and DevOps were delivering benefit in line with these two objectives at many levels. For example, when asked if they had seen ‘significant’ improvements (i.e., more than 20%) since adopting agile and DevOps, 64% of the survey respondents said they had seen improvement in the area of on-time delivery, 63% in predictability, 62% in reducing the cost of their quality activities, and 61% in customer experience.

Agile Adoption Continues to Grow

This chapter of the WQR also assesses the evolution of agile and DevOps and the tools/approaches used to assure quality. For example, we discover that although the agile implementation of packaged enterprise systems has been slow to take off, with waterfall being the predominant methodology for many years, agile adoption has started to grow. Indeed, 59% of the survey respondents now have a well-implemented agile methodology for quality and testing. Quality is being further assured by a number of different approaches, such as pre-built test case repositories for certifying sprints, which has been adopted by 63% of the survey respondents.

When it comes to enterprise systems, testing isn’t always carried out by quality engineers. Some 62% of surveyed organizations say it is carried out by business SMEs. Nonetheless the skills of quality engineers remain integral to agile teams with 32% of organizations saying quality engineers make up between 26% to 35% of their agile teams, and 28% of organizations saying their agile teams comprise even more quality engineers at between 36% to 45% of the team make-up.

Recommendations for Success

As quality orchestration increases within the software development lifecycle (SDLC), the WQR looks ahead to possible future developments, such as a need for much higher levels of automation and quality as the pace of continuous quality grows. It also makes several recommendations for ensuring agile and DevOps success across the SDLC, such as making quality engineers integral to agile development programs, and blending both technical and business skills within the broader quality engineering skillset.

Get in Touch

If you’d like to hear more about our findings relating to quality orchestration in agile and DevOps, please get in touch with:

Author

Bart Vanparys

Practice Lead, Quality Engineering & Testing, Sogeti

Bart has carried many titles in his 20 year career. He’s been an analyst, tester, quality assurance consultant, test manager, project manager, BI developer, quality manager, change manager, CoE lead, program quality lead.

A constant has been his search for ways to deliver value through IT solutions in a controlled and safe manner. He has gained experience in many domains including testing, quality management, service management, project management and architecture. He believes that a broad background combined with deep specialized expertise in selected domain is essential to be valuable in our industry.

Bart’s specialty is testing and quality engineering. His broad interest is in everything else. Bart graduated as Commercial Engineer (KU Leuven, Belgium) and has worked in IT consultancy since 2000. In 2011, he joined Capgemini Belgium where he took a lead role in building the Sogeti Testing & Quality Management practice. He has performed assignments in public sector (European Commission), finance and retail. He is currently supporting organizations in building testing & quality engineering capabilities.

Sogeti is an EXPO Gold partner at EuroSTAR 2023, join us in Antwerp

Filed Under: Agile, DevOps Tagged With: 2023, EuroSTAR Conference

Why Do Testers Need CI/CD Systems?

April 19, 2023 by Lauren Payne

Thanks to JetBrains for providing us with this blog post.

This post was originally published on the JetBrains Qodana Blog.

Competency in the TestOps field is now just as much an essential requirement for QA engineers as the ability to write automated tests. This is because of the ongoing development of CI/CD tools and the increasing number of QA engineers who work with pipelines (or the sequence of stages in the CI/CD pipeline) and implement their own.

So why is CI/CD such an excellent tool for quality control? Let’s find out!

Running Tests Automatically

Automated tests haven’t been run locally in what feels like ages. These days, CI/CD pipelines run tests automatically as one of their primary functions.

Pipeline configuration can be assigned to DevOps. But then we will be a long way from making use of the CI/CD tool’s second function: quality control, or more precisely, “quality gates”.

Quality Control Using Quality Gates

But what are quality gates? Let’s say the product code is like a castle. Every day, developers write new code – which could weaken the foundations of our castle or even poke holes in it if we are really unlucky. The purpose of a QA engineer is to test each feature and reduce the likelihood of bugs finding their way into product code. Lack of automation in the QA process could cause QA engineers to lose sleep, since there is nobody to watch over all the various metrics – especially at dangerous times, like Friday evenings when everyone wants to leave work and is hurrying to finish everything. An ill-fated merge at that moment can cause plenty of unwanted problems down the line.

This problem can be solved by building-in quality checks.

Each check deals with a different important metric. If the code doesn’t pass a check, the gates close, and the feature is not allowed to enter. A feature will only be merged into the product when it has passed all the checks and potential bugs have been fixed.

What Quality Checks can be Included in the CI/CD Pipeline?

We need to put together a list of checks to ensure that the process is as automated as possible. They can be sequenced in a “fail first” order. A feature must pass all the checks to get through the pipeline successfully. The initial checks ensure the app is capable of working: build, code style check, and static analysis.

“Build” speaks for itself: if the app fails to build, the feature does not progress. It is important to incorporate a code style check into your CI/CD pipeline to ensure the code meets unified requirements, as doing so allows you to avoid wasting time on this kind of bug during code reviews.

Static analysis is an essential tool for judging code quality. It can point out a vast number of critical errors that lead to bugs and decrease the number of routine and repetitive tasks for the QA team. Afterwards, developers should fix the detected issues and hand the code over for the testing stage.

We then continue with stage-two checks: unit tests with coverage analysis and coverage quality control, as well as integration and systems tests. Next, we review detailed reports of the results to make sure nothing was missed. At this stage, we may also perform a range of non-functional tests to check performance, convenience, security, and screenshot tests.

When developing a pipeline, we need to pay attention to 2 competing requirements:

  1. The pipeline must guarantee the best possible feature quality in light of your needs.
  2. Time spent running the pipeline should not slow down your workflow. It should generally take no more than 20 minutes.

Examples of Tools to Incorporate in Quality Checks

Code Style Highlighting

A code style is a set of rules that should be followed in every line of code in a project, from alignment rules to rules like “never use global variables”.

You might be wondering what style has to do with testers. The answer is a lot. A style check provides several benefits for QA experts, not to mention the rest of the team:

  1. A unified style helps developers work with the code and gives them more time to implement new features and fix bugs.
  2. A unified style allows you to dispense with manual code checks and use a CI/CD tool to run the checks instead.

Large companies usually have their own style guides that can be used as examples. For instance, Airbnb has a JavaScript style guide, and Google maintains several guides. You can even write your own, should you wish.

The choice of tools for code checking depends on the language. You can find a suitable tool on GitHub or find out which tools other teams use. Linters use bodies of rules and highlight code that fails to abide by them. Some examples include ktlint for Kotlin or checkstyle for Java.

Static Code Analysis

Static code analysis is a method of debugging by examining source code without executing a program. There are many different static code analyzers on the market.

We’ll now look at a platform we’re developing ourselves – Qodana. The significant advantage of this code analyzer is that it includes a number of inspections that are available in JetBrains development environments when writing code.
Many of you probably use an IDE-driven approach, where the IDE helps you write code and points out bugs such as suboptimal code usage, NullPointerExceptions, and duplicates.

But unfortunately, you can never be sure all the critical problems found by the IDE were fixed before the commit. However, you can ensure that the issues will be addressed by incorporating Qodana into your CI/CD pipeline.

Qodana, the latest addition to the family of products from JetBrains, is a cutting-edge static analysis platform designed to help developers and QA engineers improve their code quality, making it more efficient, maintainable, and bug-free. Its static analysis engine is the only solution on the market that brings native JetBrains IDE code inspections to any CI/CD pipeline. The platform provides an overview of project quality and lets you set quality targets, track progress, and automate routine tasks like code reviews.

Interactive inspection report in the Qodana code quality platform.

If you can’t fix everything at once, you can select critical problems, add them to the baseline, and gradually work your way through the technical debt. This allows you to avoid slowing down the development process while keeping the problems that have been found under control.

The updated baseline in the Qodana code quality platform.

Test Coverage

Test coverage is a metric that helps you understand how well your code has been covered by your tests (generally unit tests).

Here, you need to define the minimum coverage percentage you want to support. The code won’t be able to go live until it has been covered sufficiently by the tests. The minimum percentage is established empirically, but you should remember that even 100% coverage may not completely save your code from bugs. According to this article from Atlassian, 80% is a good figure to aim for.

Different coverage analyzers are available for other languages, such as Jacoco for Java, Istanbul for JavaScript, or Coverage.py for Python. You can build all these analyzers into your CI/CD pipeline and track the metrics with ease.

Shaping the Release Process

In addition to automatically running tests and ensuring particular code quality requirements are satisfied, the CI/CD tool lets testers organize the release process.

The release process can be complex and depend on many different manual actions. It is often a completely manual process: the artifact is created by a developer, then passed to the testers for checks, and finally comes to the person who knows how to roll it out for the go-live. Once again, there are a lot of potential choke points here. For instance, one of those people could fall ill or go on vacation.

An effective release process will look different for each team, but it will generally include the following steps:

  1. Each change in the Git branch triggers a build of the app.
  2. The build undergoes quality checks and does not become part of the main branch until it passes all the checks successfully.
  3. A release candidate is taken from the release branch or the main branch: this fixes the version and guarantees that nothing will go live unless it has been tested and has not been changed afterwards. This helps with tracking releases and all the changes they include. In addition, storing artifacts of the stable version makes it possible to revert to them quickly in the event of an unsuccessful release.
  4. The release candidate is tested and undergoes final checks.
  5. The release candidate goes live. This may be either a manual or automated pipeline launch, if the release candidate passed all the checks at the preceding stage. The choice between an automatic release process and a manual one will depend on how frequent and important the releases are, as well as the preferences among team members and the convenience of the rollout.

Any CI/CD system allows you to set up this type of process, which should be convenient for the whole team, including the testing team.

Given the factors outlined above, we believe following these basic rules will help ensure an easy and efficient release process:

  • Artifacts must be ready for download and testing, ideally stored in one place.
  • As many checks and tests as possible must be automated.
  • All complex operations with builds should be as automated as possible.
  • All builds that will go live should be recorded and remain available for a certain period after release. This will help if you need to investigate errors in the production version, reproduce bugs, or just track the history.

We would also like to remind you that if quality metrics are not controlled automatically and are not actionable, they are useless, as there’s no way to guarantee that these metrics will be adhered to.

Implement pipelines, automate processes, and use static code analysis!

Your Qodana team

Author

Alexandra Psheborovskaya, QA Lead and Product Manager at JetBrains

JetBrains is a global software company that creates professional software development tools and advanced collaboration solutions trusted by more than 12.8 million users from 220 counties and territories. Since 2000, JetBrains has built a catalog of 34 products, including PyCharm, IntelliJ IDEA, ReSharper, PhpStorm, WebStorm, Rider, YouTrack, Kotlin, and Space, a new integrated team environment.

Qodana is the code quality platform from JetBrains. It provides a project overview and lets developers and QA engineers set up quality gates, enforce project-wide and company-wide coding guidelines, better plan refactoring projects, and perform holistic license audits. Qodana’s static analysis engine enriches CI/CD pipelines with all of the smart features of JetBrains IDEs, supports 60+ languages and technologies, and allows analysis of unlimited lines of code.

JetBrains is an EXPO Platinum partner at EuroSTAR 2023, join us in Antwerp

Filed Under: DevOps, Test Automation, Uncategorized Tagged With: 2023, EuroSTAR Conference, Test Automation

Trends Software Testers Should Watch In 2023

April 12, 2023 by Lauren Payne

Thanks to Mabl for providing us with this blog post.

The digital experience has become a core aspect of the enterprise, forever linked to customer experience and business outcomes. Customers are online more than ever before, and enterprises have to offer reliable, high-quality capabilities that facilitate engaging and inclusive digital experiences. To exceed this elevated bar in 2023, software test teams should be aware of the trends in DevOps, cloud migration, and digital transformation that will continue to be enabled by strong quality practices.

DevOps Adoption Continues to Grow

DevOps redefines how teams, processes, and technology are aligned within an enterprise to make software development faster and more collaborative through automation. According to the 2022 Testing in DevOps Report, enterprises of all sizes are pursuing full DevOps adoption with automated workflow pipelines. Embracing DevOps practices successfully requires a continuous view of quality, which requires elevated, automated testing.

DevOps pays off. Google’s 2022 Accelerate State of DevOps Report reveals that high-performing software teams are more likely to embrace DevOps practices like version control (33%), continuous integration (39%), and continuous delivery (46%).

Teams leveraging these practices are quickly maturing in a way that improves overall product velocity, and overall application quality as well. Mature organizations where quality is at the center of their DevOps transformation will achieve these DORA metrics faster while delighting customers with the frequent delivery of new battle-tested features.

Adoption of Cloud Technologies is Accelerating

With the massive shift to cloud-based architecture, software testers must be mindful of controlling quality throughout the migration. Often, the migration to the cloud goes together with DevOps adoption. As development cycles increase, QA teams should investigate cloud testing solutions to quickly scale up test execution – without the limitations of managing on-premise infrastructure.

More Reliance on APIs

Enterprises are becoming more connected and integrated – all possible via APIs. As with cloud computing, this makes the infrastructure more dynamic. It also presents the problem of having to contend with the quality of third-party APIs that may not be properly updated and secure and determining how it will affect the status and quality of the enterprise’s applications. Prioritizing quality with continuous API testing for integrated services allows test teams to detect issues before they affect customers.

Digital Transformation Initiatives are Increasing

Digital transformation remains a top priority for enterprises. In fact, according to one survey, 94% of CFOs recognize the need to maintain or accelerate the already-intense pace of transformation incited by the pandemic. To continue the digital transformation momentum, quality must become a foundation of software development and delivery.

To help enable digital transformation, QA teams can embed quality throughout the software development lifecycle while finetuning processes and QA support. Outside of ensuring the functional correctness of your application, QA teams can start embedding end-to-end, performance, and accessibility tests in the pipeline as well. This way, you confidently deliver better experiences for customers throughout your organization’s transformation.

Ensure Great CX with Low-code Test Automation

Quality is the differentiating factor among competing organizations. Software QA teams have the opportunity to transform technologies, develop new processes, and contribute to excellent customer experiences if they are successful in building a culture of quality in their organization during transformational times.

Once you’ve built a culture of quality and set up the right team and processes, QA leaders need the right tools to execute. These transformations are enabled by intelligent, low-code test automation like mabl. Mabl allows anyone, regardless of coding experience, to create, execute, and maintain tests, integrate those tests into your development pipeline, and share insights into the holistic quality of your application back to the entire software team.

Mabl helps teams scale their testing with 90% less effort, giving QA teams time back to grow test coverage and focus on testing activities that positively impact the customer experience.

Author

Leah Pemberton

Director, Marketing at Mabl

Leah is the Director of Marketing at  Mabl, writing frequently about quality engineering and building a culture of quality.

 Mabl is an EXPO Gold partner at EuroSTAR 2023, join us in Antwerp

Filed Under: DevOps, Software Testing, Test Automation Tagged With: 2023, DevOps, EuroSTAR Conference

How 2023 IT Trends Impact Software Testing

March 17, 2023 by Lauren Payne

Thanks to OpenText / MicroFocus for providing us with this blog post.

[Disclaimer: The future described in this article is the author’s personal view and is by no means any indication of OpenText products’ direction.]

Happy 2023 to the community! The beginning of a year is always the time to think big and look into the future. Last year, I wrote about three 2022 trends that’ll change test management, all of which continue to gain momentum. So far, you’ve heard 2023 predictions from analyst firms, consulting firms, and others. Among those trending IT keywords, I picked a few important ones to share with you. These words may not be new. But they have become even more important in today’s challenging business environment. Putting them together, we can see new ways of developing, testing, and measuring software.

Industry Cloud Booms

Gartner defines industry cloud platforms as modular, composable platforms supported by a catalogue of industry-specific packaged business capabilities. Packaged business suites are not new. The ERP suites from SAP, Oracle, and other vendors are well-known examples. Industry cloud platforms offer a rich set of industry-specific capabilities from many vendors and a place to compose and deliver business applications faster. Enterprise integration PaaS gives you the tools to easily integrate those industry components to form your application with low code or no code.

Cloud is the marketplace of other good stuff too, including AI and data analytics capabilities. Container technologies let you deploy your applications easier and swap out a defective module quickly.

Leveraging the industry cloud shortens the cycle to deliver value and lets you focus on innovation instead of the underlying mundane tasks. Some industry cloud platforms are already full-fledged, including the SAP Industry Cloud and Microsoft Cloud for industries. Companies with deep industry knowledge, like Deloitte, also join the play with their industry solutions.

When your application is built in this way, a focus of testing is verifying the business processes—the “wiring” you composed—and the testers are likely business users. Tools to efficiently verify business processes will find many new places to shine. Testers of all skill levels will appreciate the ability to manage complex interdependencies and save efforts with a component- or model-based approach. Model-based testing directly generates test cases from your business process model and automates them. Expect it to become more common in 2023.

With cloud platforms for building applications fast, you won’t want to lose momentum in the testing part. That’s why cloud-based testing and device farm are your choice.


Observability-Driven Development

The concept of observability has been around for decades. But it recently got popular in the cloud world because it allows people to observe the health of cloud-based systems by analyzing logs and correlating events. Now observability has come to software too, with observability-driven development (ODD). By adding a small amount of code for instrumentation, the development team can observe what’s happening when the application runs in the production environment in the cloud. When there is any issue with the app, observability helps identify the cause quickly and accurately—pinpointing the specific code in question.

ODD is yet another way to strengthen the feedback loop from Ops to Dev. ODD is also said to be a way to test in production, i.e., shift-right testing. The biggest advantage of testing in production is that you have real data and the same environment as customers use. Such observability is also valuable when you test in staging environments. We’ll see more test management or DevOps governance tools incorporate observability data into their data repository for analytics and insights in 2023.

FinOps Meets Value Stream Management

The word FinOps combines “Finance” and “DevOps,” stressing collaboration between business and engineering teams.

FinOps started when cloud service providers and other vendors wanted to provide a way of managing finances for a company’s cloud-based systems. It’s meant to help organizations find the best spending model for achieving their business goals. OpenText HCMX supports FinOps.

I expect to see FinOps and value stream management (VSM) converge sometime in the future. Both are extensions of DevOps. FinOps focuses on managing your costs against business value, while VSM focuses on eliminating bottlenecks and wastes during the process of delivering business value. With both sets of data, you can optimize the cost performance of your application. For example, when you’re unsure whether it’s more efficient to outsource a certain part—such as testing—FinOps and VSM data will likely give you a clear answer.

What’s Next?

You may have noticed that all of the above are related to the cloud. Why? Challenges in the global economy are driving more cloud adoption. The cloud has become the frontier of innovation. Every enterprise that wants to win and lead must embrace the cloud. OpenText software quality solutions went the SaaS model to serve customers who are moving to the cloud in two ways—reducing tooling cost and increasing speed.

Stay tuned on what’s happening with OpenText cloud solutions, especially the ValueEdge platform, in which you will find the most innovative software quality capabilities from OpenText at the earliest.


What is your organization’s cloud initiative? We are here to help. Contact us to discuss details.

Author

Ying Lei

Ying Lei is a senior product marketing manager at OpenText who specializes in test management, application lifecycle management, DevOps and value stream management.

OpenText is an EXPO Gold partner at EuroSTAR 2023, join us in Antwerp

Filed Under: Agile, DevOps, Software Testing Tagged With: 2023 trends, agile, ALM, Cloud Testing, DevOps, SaaS, software testing, Test Management, Value Stream Management

3 Obstacles to Continuous Testing & How to Remove Them

July 6, 2021 by Fiona Nic Dhonnacha

Continuous testing is a process that empowers teams to build quality into software development and accelerate the delivery of high-quality customer experiences. With continuous testing, teams get instant feedback on code health using automated testing.

Continuous testing allows organizations to assess business risk. Recent industry surveys show the top metrics used to track project progress and success:

  • High test coverage
  • Increased defect remediation
  • Decreased defects in production

Building Quality Into the Development Process

Quality is an important topic for senior levels of management. Here are some insightful findings of business leaders polled:

  • 48% said quality improvement was in their top three initiatives.
  • 68% were seeking ways to improve delivery speed and quality.

The new goal? Build quality into the development process to accelerate the delivery of high-quality customer experiences.

How is this combination of speed and quality achieved? Continuous testing. But it does come with its challenges.

Overcoming Continuous Testing Obstacles

With quality at speed as the goal, there are typically three primary obstacles to overcome when implementing continuous testing.

  • Lack of expertise. The team may lack in availability to take on new approaches and skills needed to learn and adopt new tools and techniques.
  • Unstable execution. Test automation as it currently stands in the organization may be unstable and unreliable. As code grows, so does execution time.
  • Unavailable environment. The test environment is often unavailable, uncontrollable, and constrained by system dependencies.

Teams must remove these obstacles for continuous testing to become ingrained in the development culture of a software organization.

Obstacle 1: Lack of Expertise on the Team

Initially, the lack of expertise isn’t only the team’s lack of knowledge and skills. It’s also limitations of tools being used.

Consider user interface (UI) test automation. It’s a common practice and reliable, but reusable automation is difficult. Selenium is the de facto standard. While open source and free, it has its own adoption curve, and it takes experience and time to master.

Selenium tests can be unreliable and what’s recorded one day, can’t be played back the next. Test maintenance becomes a growing issue as more UI tests are automated. Selenium requires further tools support to become easier to use and maintain.

Service level or API testing is a relatively new, but valuable practice. However, it sits in a no man’s land between developers and testers. Developers understand the APIs the best but aren’t motivated nor compelled to test them and testers lack the knowledge needed to do API testing.

The number one challenge when organizations are trying to adopt API testing, is understanding how the APIs are actually used. This isn’t to say APIs aren’t documented or designed well. Rather, there isn’t much information about how the services are being used together in a use case, workflow, or scenario.
lack
In addition, it’s important that API test automation goes beyond record (during operation) and playback (for testing). Modeling the behavior and the interactions between APIs are needed, as is using these interactions to steer test creation and management processes.

Performance testing is often seen as something that’s done by another team in the organization, perhaps performed as a check box item. But when performance issues arise, product development may have moved forward, and a disruptive rollback may be required.

Ideally, performance testing needs to be done earlier in the software development process and leverage the work already being done with automated functional testing. At the same time, the team is adopting API testing, they can leverage that work to enable performance testing to shift left, making it a joint responsibility of developers and testers.

How to Remove: Simplify Test Automation

The lack of expertise and training in the development and test team should not reflect on the team itself, but rather on the complexity when adopting test automation and associated tools. There are solutions available that strive to simplify test automation. These solutions make adoption less disruptive and integrate better into existing processes.

Create reusable, maintainable, and understandable test scripts. Parasoft Selenic solves the main issues with Selenium adoption: test creation and maintenance. By recording UI interactions via the Chrome browser, Selenium test cases are automatically created based on these interactions. In addition, locators are recorded using the page object model to be more resilient to changes in the UI. Selenic uses AI-driven self-healing of tests so that when UI changes break existing tests, the tool makes intelligent assumptions to prevent the test cases from failing.

Model real-world API test scenarios by recording manual and automated UI interactions. API test adoption is hindered by the ability to create tests. Parasoft SOAtest uses existing UI tests (including Selenic-created tests) to record API interaction happening during execution of the application. AI within SOAtest organizes these recorded interactions into recognizable scenarios which then form the basis of an API test repository. These API scenarios can be played back, edited, cloned, and reused to form a comprehensive API test suite. The automation and AI-powered decision making SOAtest does to make API testing easier to adopt, use, and maintain. Plus, it helps bridge the API testing knowledge gap in the development team.

Reuse existing test artifacts to efficiently scale load, performance, and security testing as part of DevOps pipeline. As the development team becomes more proficient at test automation for the UI and API levels, the test repository becomes an important reusable resource. Tests can be reused for load and performance testing and to increase use case and code coverage.

The process flow shows automation for Selenium UI tests, recording and creation of API tests, and reuse of assets for future functional, performance, and security tests.

Obstacle 2: Execution of Tests Is Unstable, Unreliable, & Takes Too Long to Run

Understandably, software organizations expect automated tests to be efficient and not impede development progress. However, as test suites grow, so do the problems with maintaining and executing them. Tests, like code, are impacted by change. New functionality added during a sprint can significantly impact the user interface or the workflows of the application. These changes break existing tests, making them unstable. It’s important to address those as quickly as possible.

When tests fail, you need to understand the context of the failure. Not every test failure is equal. Some use cases are more important than others or, perhaps, some tests are unstable by nature. What is lacking is an understanding of the impact of test failures or test instabilities on the business priorities of the application. Investigating these constant test failures becomes a distraction from the overall test automation strategy. Correlation between business requirements and tests is critical to ensure that the value of automation is realized.

Another obstacle is the actual execution time for test suites. As the test portfolio grows, so does the execution time beyond a reasonable waiting period for feedback. Quick feedback to change is essential for a successful CI/CD pipeline so test efficiency and focus are required.

How to Remove: AI-powered Test Execution

The solution to the test execution obstacle is to test smarter with AI. This means leveraging test automation AI to make tests more resilient to change and to target execution on key tests only.

Smart UI testing with Selenium and Selenic. Using AI, Parasoft Selenic self-heals tests when UI changes are detected. These are automatically used but recommendations are sent to the developer to help fix the tests. These fixes can be automatically applied to the Selenium tests, removing manual debug and code changes.

Using AI to self-heal tests reduces test instability but also provides recommendations and quick fixes.

Plan work item tests based on impacted requirements. To prioritize test activities, correlation from tests to business requirements is required. Keeping track of user stories and requirements provides real-time visibility into the quality of the value stream. User stories and requirements should be reviewed for priority. The traceability capabilities in Parasoft SOAtest are used to plan execution for tests that validate items being worked on in-sprint. However, more is required since it’s unclear how recent changes have impacted code.

Prioritizing tests based on impacted user stories is the first step to optimizing test execution.

Use test impact analysis to validate only what has changed. To fully optimize test execution, it’s necessary to understand the code that each test covers, then determine the code that has changed. Parasoft tools provide this capability through a central repository for test results and analysis. Test impact analysis allows testers to focus only on the tests that validate the changes.

Test impact analysis determines which tests correlate to the code that changed to focus testing only on what should be tested.

Obstacle 3: Test Environment Is Unavailable, Uncontrollable, & Constrained by System Dependencies

The testing environment is the linchpin of the obstacles stopping organizations from turning automated into continuous testing. There are three types of challenges that organizations face when trying to make tests run anytime, anywhere, and dealing with the external dependencies of the application. This is especially true for a microservices architecture. The number of dependencies explodes due to the very nature of the design.

Test Environment Challenges

Waiting for access to a shared system, like a mainframe or an external dependency provided by a third party. Availability might be time limited and costly. It’s also a challenge if the external dependency is heavily loaded with multiple people working on it at the same time, resulting in test instability from data collisions.

Bottlenecks caused by delayed access. This is due to the nature of parallel development and typical of modern processes. For example, multiple teams are collaborating to deliver new features to the value stream such as interdependent microservices. Testing can’t proceed on one microservice because another isn’t available yet.

Uncontrollable test data. Although microservices are relatively easy to deploy and test in isolation, their dependencies on data or performance characteristics limit the ability of them to be tested thoroughly. For example, reliance on data in a shared production database can limit the ability to test services.

How to Remove: Control the Test Environment

Start simulating these dependencies to give the team full control using a service virtualization solution. Parasoft Virtualize simulates services that are out of your control or unavailable. It provides workflows that:

  • Enable users to access complete and realistic test environments.
  • Stabilize their test environment.
  • Get access to otherwise inaccessible dependencies.
  • Manage the complex business logic, test data, and performance characteristics required for virtual services to behave just like real services in the real environment they represent.

Service virtualization removes the bottlenecks. Here’s how.

Record and simulate: Capture, model, and provision simulations of live systems.

Using the recording capability of Parasoft SOAtest, it’s possible to capture the behavior of the application in its environment. Parasoft Virtualize models the behavior of external dependencies making it possible to remove and simulate the behavior of dependencies, dynamically on the fly, switching out real versus virtual. Making these services and dependencies available and stable, virtually, accelerates the testing process and enables continuous testing.

Dependencies in the test environment are obstacles to testing. Virtualizing these dependencies removes their impact on testing and enables continuous testing.

Deliver a prototype first: Model behavior based on contract descriptions or payload examples.

Service virtualization enables prototype development based on the contract descriptions derived from the API interaction recordings and analysis in SOAtest.

Dependent services can be simulated with good fidelity to create prototype versions that satisfy their roles in the system when testing another adjacent service. This removes the schedule limitation inherent in parallel development—even when services aren’t complete, they can be virtualized for testing other services.

Environment Manager provides graphical modeling and control of the test environment. Services are controlled and virtualization parameters are configured using this diagram.

Synthesize private test data.

Another obstacle in testing enterprise applications is test data. Many organizations use real data, but this is fraught with privacy concerns. Purely synthetic data is often not realistic enough to test with so a compromise is needed. Synthesizing real data by removing personally identifiable information (PII) provides realistic and safe-to-use data. Test data management is required in conjunction with service virtualization to provide a realistic, highly available test environment that won’t result in any privacy compromises.

The Benefits of Continuous Testing

Removing the key obstacles to continuous testing enables testing to occur on a regular, predictable schedule. It transforms application testing empowering teams to:

  • Test earlier. Shift left to in-sprint testing where it’s quicker, cheaper, and easier to remediate the problem.
  • Test faster. Automate and execute continuously to get immediate feedback when defects are introduced.
  • Test less. Focus and spend less time creating, maintaining, and executing test scenarios. Reduce the cost of test infrastructure.

Want to learn some more about continuous testing? Join us at EuroSTAR in September, for 3 days of talks, tutorials, and lots more from leading test experts. Check out the full programme.

Parasoft is exhibiting at EuroSTAR Conference this year – take a look at their work here.

Author

Mark Lambert, VP of Strategic Initiatives at Parasoft

Mark focuses on identifying and developing testing solutions and strategic partnerships for targeted industry verticals to enable clients to accelerate the successful delivery of high quality, secure, and compliant software. Since joining Parasoft in 2004, Lambert has held several positions, including VP of Professional Services and VP of Products. Lambert is a public speaker and author. He’s been invited to speak at industry events such as JavaOne, Embedded World, AgileDevDays, and StarEast/StarWest. He has published thought-leadership articles in SDTimes, DZone, QAFinancial, and Software Test & Performance. Lambert earned both his Bachelor’s and Master’s degrees in Computer Science at Manchester University, UK.

Filed Under: DevOps, EuroSTAR Conference Tagged With: DevOps

Understanding the DevTestOps Journey

March 24, 2021 by Suzanne Meade

Understanding the DevTestOps Journey

For the last two years, our team has hosted the Testing in DevOps Landscape Survey to take the pulse on how quality testing is evolving as adoption of DevOps grows. We turn to our dedicated community and the software development industry at large to understand how quality engineering is shaping the adoption of DevOps practices, how they champion the customer experience in a rapid development framework, and what obstacles remain in the way.

Last year, an incredible 1,030 people with experience in QA management, testing, and development shared their insights. Their expertise resulted in an in-depth look into how test automation is elevating the voice of quality in a DevOps world. We hope you will help us again this year to collect the data we need to understand the current state and what’s changed over the past year. We are especially looking forward to learning about how teams are adapting to the global pandemic and the impact it’s had on their digital transformation objectives.

In case you missed it, the 2020 DevTestOps Landscape Survey found that most participants were making good progress in their DevOps adoption journey, with 31% saying they were agile and 35% saying they had made strides towards DevOps. Almost one-fifth described themselves as “so DevOps it hurts.” What’s more, the 2020 survey revealed the close connection between DevOps adoption and customer satisfaction:

DevOps chart

This year, we are increasing the focus on solving the real-world challenges of DevOps implementation, especially in a virtual world. With the customer experience front-and-center post-2020, we want to understand how DevOps practitioners are rethinking long-term quality and testing strategies, where testing takes place in the DevOps pipeline, and what the overall QA impact is on the customer. We also want to dig deeper into what challenges remain and what’s top of mind as we head into 2021.

Your experience is valuable to us! Please take a few minutes to fill out the survey today. With your help, we look forward to learning and sharing these important benchmarks and insights early this summer.

mabl are Gold Sponsors at the 2021 EuroSTAR Software Testing conference, which takes place 28-30 September this year. The programme will be launched soon, so stay tuned!

Izzy Azeri, mabl co-founder

Author: Izzy Azeri

Izzy Azeri is the co-founder of mabl, the leading test automation tool built for DevOps. He co-founded mabl in 2017 with Dan Belcher after realizing that existing software testing tools were unable to support the needs of high-velocity development teams. Mabl is now the leading intelligent test automation tool for quality-centric brands like Ritual, Charles Schwab, and Arch Insurance. The company has raised over $36 million from prominent VC firms such as GV (formerly Google Ventures), Amplify Partners, CRV, and Presidio Ventures.

Filed Under: DevOps Tagged With: DevOps

EuroSTAR Huddle Deep Dive: Agile

March 11, 2021 by Ronan Healy

It has been twenty years since seventeen software developers met at a resort in Snowbird, Utah and since then Agile has become the dominant form of developing software.  At EuroSTAR Huddle, we will be acknowledging this with our just announced EuroSTAR Huddle Deep Dive: Agile week. Taking place from 22-26 March, the EuroSTAR Huddle Deep Dive will feature seven speakers, all experts on Agile who will guide you through all things Agile.

So at EuroSTAR Huddle we have invited seven highly regarded experts on Agile and software testing. Over the course of one week they will share their expertise with you across seven live events.

During the week we have some of the best minds on Agile. We have two new events in two AMA (Ask Me Anything) sessions.  First off is Lisa Crispin, a leading expert and author of numerous Agile books. She will be taking part in an  on Testing in Continuous Delivery. Also taking part in another AMA is Bob Galen who will be exploring Testing in Agile Context. Both events give you the chance to put your questions to these Agile experts with a wealth of experience in Agile.

But there is more! We will also have live presented events with practical takeaways that you can apply with your team. Gitte Ottosen will be sharing her thoughts on Quality in Agile and in particular what it means for your customer. Derk-Jan de Grood will also explore quality focusing on built in quality and how quality is organised at the team level. Speaking of teams in Agile Selena Delesie will be joining the Deep Dive week to demonstrate how you can super-charge your agile team and create Agile super teams. Mia Johannsson will explore the four key cornerstones of Agile QA and its importance in the software development process. Finally moving from teams to the user level, Hanna Dernbrant is going to explore what it is like to move from Waterfall to DevOps and what experiencing that sort of change can be like.

With this great quality (no pun intended) of speakers and minds on Agile, the EuroSTAR Huddle Deep Dive: Agile week is not to be missed. Plus there will be blogs throughout the week on all things Agile. Make sure to join us there.

Filed Under: Agile, DevOps Tagged With: agile

Continuous Testing Is Not Automation

September 2, 2020 by Fiona Nic Dhonnacha

Many people confuse continuous testing with test automation. That makes sense, because you cannot do continuous testing without automated tests. But, it’s much more.

There are four basic questions you can ask to see if your automation is truly continuous:

  1. Are your tests automated in real time?
  2. Are they included as part of your continuous integration (CI) pipeline?
  3. Do you allow them, alone, to determine that a version of code can move to the next step in the process?
  4. Are performance and security tests included in questions 1,2, and 3?

If you cannot say yes to all four questions, then you are not doing continuous testing.

So, how do you go from automated tests to continuous testing?

First, try to run your test suite back to back. If it takes more than a minute or two for each test to run, you need to look at the tools you are using for automation. Are you able to automate in real time without a complex abstraction layer for the testers? Are you using a third party that is inherently slower than an open source tool? Your answers might indicate that your automation tools do not support continuous testing.

continuous testing in automationThe other reason tests could be slow is if they all originated from your regression suite and thus are all end-to-end UI tests. In this case, you will need to adjust how you have applied the testing pyramid to ensure you reduce the number of tests, so that more is done at the API and component level.

Second, look at the failed tests to see if different tests failed in both runs. If a test passes in one run and fails in the next, then you might have flaky tests, which again can be solved by refining your testing approach, as well as evaluating your testing framework for good coding practices.

Flaky tests could also be caused by dependencies on data or downstream systems that are not reliable. In these cases, you should pull out all the data conditioning into its own suite of tests, or find a more reliable way to inject data, like via a test data management tool or through APIs directly. You also might need to leverage a service virtualization tool to replace unreliable or unavailable systems.

Third, integrate the suite with your CI tool. You could start with deconstructing your suite of tests so that you have multiple suites running at different points of the process, each building on so that coverage increases. You will also need a dashboard so teams can see the results of the tests and gain confidence that a failed test is truly a defect.

Finally, integrate performance and security tests into this process. You can follow the same steps as above, because the tool, approach, and CI integration may all have to be addressed.

Continuous testing has a higher-level maturity than automated testing that could require a totally different way of working. The result is a fast feedback loop for developers—and a faster path to production.

Learn more about continuous testing and automation at EuroSTAR Online this November:

 

From Rags to Riches: Turning Your Test Automation Into a Cinderella Story

Surviving and Thriving in the Automation Jungle

Continuous Delivery in 4 months for 15 Teams and their 1 Monolith

——————————————————————————————–

Epam blog author imageAuthor: Adam Auerbach, VP, Co-Head of Cloud and DevTestSecOps Practice at EPAM Systems

 

Filed Under: DevOps, Test Automation

  • Page 1
  • Page 2
  • Next Page »
  • Code of Conduct
  • Privacy Policy
  • T&C
  • Media Partners
  • Contact Us