Bloggo back to the blog
Test automation – the use of morning availability verification of environments (based on example of a project for Orange)-->
Case study: Common environmental errors causing stoppage of testing works in the morning.
It happens very often that night or early morning code migration and application servers restarts cause errors on the environment, which suspends testing works. There are about 100 manual testers that can be blocked by environmental and system defects.
Environment we are working on: over 30 integrated systems. Technologies and systems mainly used:
– WEB (HTML, Ajax, jScript)
– PeopleTools, PeopleCode
The simplified schema (1) below shows how the environment looks like. However, in reality there are many more systems and the communication between them is more complex.
Solution: We came up with the idea of using automated testing for morning verification. We have selected 14 key business processes that we decided to verify every day.
Our framework consists of reusable action container, function library, object repository and test data sheet. We have also created a batch file which can be used to schedule an execution of test scripts.
– Reusable action container – pack of actions that can be used in many scripts (for example logging into application)
– Function library – library containing our external functions (for example function checking serial numbers for devices ordered during some business processes)
– Object repository – we share objects from all scripts in one repository (helps in reusing some actions in another scripts)
– Test data sheet – excel file containing environmental data (for example login, password and IP address of a web service), moreover that excel contains data specific for each script.
2 Average time of running a scripts
As you can see in the chart above (2) the average time to run a script is a little shorter for automated scripts, while the maximum time is not significantly different (due to a long timeouts for communication between systems – for example, DataBase and OrderManagement). Although, our scripts verify the processes in systems directly and indirectly (many checkpoints as shown in chart 1). However, the biggest advantage of using automation is that one user can run multiple scripts simultaneously. We run our scripts on desktop computers and virtual machines. One user runs up to 6 scripts at a time. Because the processes are complex and analysis of errors can take some time there are two testers to come for morning verification. If there are errors, after analysis, they are reported in the Quality Center to the appropriate vendor (system). The report is ready before 8:00 am containing the status of the morning verification run for each script and information if necessary (defect number and system responsible for the defect).
This report is sent to the leaders of tests. Suppliers have time to resolve defects, so as not to delay the start of testers’ work. If necessary, the verification environment can be repeated during the day at any time.
Benefits: We have gained a tool to perform a quick verification of the environments and check key business processes functionalities that might be unintentionally blocked (by too early or incorrectly uploaded CR). Thanks to our solution, our client saves money on testing – testing process was shortened due to not stopping manual testers by environmental defects.
What can still be improved: We are working to make the morning verification fully automatic – scripts should not only be automatically run at a designated time but also defects should be automatically reported to the QC by our scripts. The difficulty lies in the fact that the scripts themselves are unable to clearly determine which provider (system) to report the defect. However, we hope that soon we will overcome this problem.
Jakub Eliasz is a test automation specialist at Outbox. Jakub works in the field of quality assurance since 2007, with backgrounds in informatics and business analysis he moved into test automation. Thanks to his technical expertise and business experience, he not only creates and maintains test scripts but also seeks and finds additional solutions how to improve processes. Jakub is a co-author of “morning environment verification” solution and he contributed in creation of our own test automation framework based on VBScript. Apart from mastering automation tricks and QA process, Jakub masters martial arts and develops his creativity by engaging in computer graphics and in photography.