Thanks to Testory for providing us with this blog post.
Testing has always been a bottleneck in the development process. Since product teams often sacrifice time spent testing, the workload testers face ebbs and flows.
Your company’s testers most likely know what it’s like to work weekends and evenings when there’s a release coming up. At points like those, they generally have to take on low-level work to make sure they check everything and deliver a high-quality product. But that overworks them and leads to burnout.
Product teams often think about the silver bullet: how do you scale testing (increase capacity) instantly without just throwing money at the problem?
Before we answer that question, however, we should take a step back and look at the big picture. What challenges are inherent to testing?
Testing requirements by role
| CTO | Product manager | Head of testing | |
| Faster time to market | Yes | Yes | – |
| Budget optimization | Yes | Yes | – |
| Product Quality for Customers | Yes | Yes | Yes |
| Peak loads | – | – | Yes |
| Routine tasks | – | – | Yes |
| Variety of testing enviroments | – | – | Yes |
Every role has its own problems. How do you solve them all at the same time?
A few years ago, we took a systematic approach to testing challenges, eventually coming up with a product for the largest IT company in our region. The solution married a variety of ML and other algorithms with traditional IT tools (Tracker, Wiki, TMS) and thousands of performers scattered across different time zones. That eliminated the bottleneck. With a dozen product teams online, they could scale testing or remove it altogether based on need.
On the one hand, we’re constantly improving our algorithms to give better feedback faster. On the other, our automated system selects professional testers who guarantee that same great result.
Another advantage our system offers is that it stands up well to load spikes around the clock rather than just during regular working hours.
Let’s look at an example. In February 2023, a large customer handed Testory a process that included 2240 hours of work, 1321 of which were outside business hours.
As you can see on the graph, the load placed on testers was anything but even. There are a thousand reasons why that could be. Some peaks outpaced the capacity of a full-time team working regular hours, though expanding the team would have resulted in team members sitting around the rest of the time.
All that makes sense on the graph. The red line represents hours, with eight full-time employees sufficient to cover the total of 65. As you can see on the graph, the load was more frequently heavier, meaning that team of eight wouldn’t be up to the task, though there were also times were they wouldn’t have had enough work.
How does it work?
The customer embeds crowd testing in their development pipeline, calling the process from their TMS as needed and running regress testing in our product with external testers.
When they submit work for crowd testing, our algorithms scour our pool to select the best performers in terms of knowledge, speed, and availability, then distributing tasks so we can complete a thorough product test in the shortest possible time. We then double-check the result, compile a report, and send the report to the customer. That’s how we fit N hours of work into N/X hours.
The customer can scale up testing whenever they want, then scaling back and paying nothing when they don’t have work to do. It’s an on-demand service.
Performers enjoy an endless stream of work that’s perfect for their skill set in addition to some that pushes them to learn and grow. For our part, we offer testers special skill- and knowledge-based courses, stable payment that depends on how many tasks they complete, and the opportunity to work from anywhere in the world.
What’s the bottom line?
We free up resources our clients can rededicate toward interesting and higher-risk work, help out with peak loads, and streamline costs:
How can you get that for yourself?
Testory is a separate process and product born to help large companies. It’s for anyone trying to quickly deliver IT products that solve user problems. If you’re interested in leveraging our experience, get in touch, and we’ll build a roadmap for you.
Author
Mary Zakharova
Mary has been working with crowdtesting products for 6 years. She started her career as a community manager in a testers’ network.
In recent years, Mary has been in charge of the Testory product
Testory is an EXPO Exhibitor partner at EuroSTAR 2023
