End-to-end test automation has been generally avoided for what were once good reasons. But today’s technology allows us to automate end-to-end tests effectively and without causing too much overhead.
Why end-to-end testing was originally avoided
Automating end-to-end tests is a tricky business. Many leading practitioners over the years have advised against doing ANY end-to-end test automation. They say, do a good job of automation at the lower levels, and do end-to-end testing manually once and trust your lower level tests to protect against regressions.
As a result of this, we often see the test automation triangle pictured below. The shape of the test automation triangle is predicated on traditional associations about the cost of troubleshooting test failures and feedback cycles at each level. Traditionally, the closer you were to the bottom, the easier it was to find the root cause of the issue. Therefore, they recommend that you 70/20/10% split between unit, integration, and end-to-end tests, respectively.
But no matter how much testing you do at lower levels of granularity (unit, integration, etc.) the complete end-to-end is the only thing that brings together all the pieces and configurations while focusing on the true end-user functionality and experience.
No matter how much granular testing and tracing is done at the unit and integration level, the fundamental issue is that we’re spending less time on the user experience, and more time testing how we hope the application works. We’re losing the ability to surface problems to the tester in an easy-to-understand fashion.
Cindy Sridharan makes it clear in her story about distributed tracing. The service graph she draws below depicts dependencies well, but when it comes down to debugging a scenario where a specific service is experiencing issues, such a granular view isn’t particularly useful. If there is a slowdown with the front page, there are many dependencies you’d need to start digging into to find the root cause.
Source: https://medium.com/@copyconstruct/distributed-tracing-weve-been-doing-it-wrong-39fc92a857df
Putting more focus on unit or integration testing puts us into the same scenario. From the end-user perspective, we lack test coverage and it becomes harder to troubleshoot issues when tracing issues from the user-experience point of view. So the case to be made is – does today’s technology allow us to automate end-to-end tests effectively and without causing too much overhead?
mabl’s part of a wave that can change the shape of the automated testing triangle.
Times have changed
Unit and integration tests just aren’t aligned with the user experience. Additionally, there is a class of manual tests that were reserved for manual execution because they were too difficult to automate, not because they truly required human observational powers and reasoning.
So the questions we’ve asked ourselves is: Why avoid automating end-to-end tests to avoid the problems with it? If the end-user is as important as we preach they are, why not fix the problems with end-to-end test automation so we can do it better? Faster? Reliably? Easily?
With mabl’s ease of creation of robust end-to-end tests and rapid cloud execution providing quick feedback, we’re removing the constraints that suggest the end-to-end tests should be a taper. mabl also uses big data and machine learning to lower that pain, and we do this in a way where we aren’t intrusive. We create machine learning models from the tons of test output that already exists as the tests are executed, learn from the data, and surface insights and predictions about the quality of the application under test which ranges from functional test failures to more nuanced issues like visual changes and performance slowdowns.
By eliminating the cost and difficulty of end-to-end test automation, we allow the top of the pyramid to broaden, eventually perhaps becoming a testing hourglass, with the end-goal of benefiting the end-user experience.
The bottom line is, end-to-end testing has become stigmatized because of reasons that no longer apply with the innovation we’ve seen in test automation offerings in recent years. But end-to-end tests are way too valuable to avoid doing because they best align with the user experience.
So if the recurring argument is that end-to-end testing is too expensive… what if we just made it cheap?
About the Author:
Chou Yang
Chou is an engineer turned marketer who originally studied art. She enjoys her days at mabl engaging with the testing community via meet-ups in Boston, via Twitter, and on the mabl blog. You can find her napping in random nooks in the mabl office, and at [email protected]