Bloggo back to the blog
User Acceptance Test Essentials: Early Customer Review-->
In engineering and its various sub-disciplines, acceptance testing is a test conducted to determine if the requirements of a specification or contract are met. UAT is the last step performed after the system tests are completed and before the product sent to production so as to determine if the demands of the customer are met. User Acceptance Testing (UAT) is a process to confirm that a system fulfills mutually agreed requirements.  While system testing checks whether the specified system has been delivered, acceptance testing checks whether the system delivers what was requested.
The aim of this paper is to share the benefits of using Early Customer Review in UAT (User Acceptance Test – Qualification Testing) and to explain the outcomes of proposals made for problems experienced in User Acceptance Testing.
Acceptance tests are black box system tests. Each acceptance test represents some expected result from the system. Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which tests have a higher ratio on failing to complete the process. Acceptance tests are also used as regression tests prior to a production release. 
If demanded, Early Customer Review can support User Acceptance Test while the product is at development phase. Before User Acceptance Test phase, performing Early Customer Reviews for completed phases would support User Acceptance Test. Early Customer Review process is added to WaterFall methodology in Turkcell. Thus, quality improvement and customer satisfaction are ensured by involving customer into our Software development life cycle process as in AGILE methodology.
It is performed through the participation of Project manager, demanding person (user or customer), analyst, developer and test engineer. These participants of User Acceptance Test firstly participates in Early Customer Review. It is the phase where the users for the first time meet the requirements they demand. There is no documentation for Early Customer Review. Issues are reported as “Design Issue” at this phase. If compared with User Acceptance Test, it is the phase where greater impacts are achieved with smaller touches. Thereby, it is assured that customers do not to meet any surprise in User Acceptance Test and are involved in every step of the development.
During User Acceptance Test, the client evaluates the application to ensure that it meets the documented acceptance criteria and determines if the application is ready to be implemented in production. The client evaluates a variety of application areas by typically using business scenarios; but exception tests may also be run. 
The User Acceptance Test plan is extracted and issued when the project plan is developed. User Acceptance Test analyzes cases that include all requirements specified at the demand level. The faults detected in User Acceptance Test are reported as “User Acceptance Test issue”. In User Acceptance Test, not only “requirement coverage” is surveyed, but also errors are found via “dirty testing” method. UAT test cases, scenarios and their prints are documented.
A single sampling plan for attributes is a statistical method by which the lot is accepted or rejected on the basis of one sample. Suppose that we have a lot of size ; a random sample of size is selected from the lot; and an acceptance number is determined. If it is found the number of nonconforming is less than or equal to , the lot is accepted; and if the number of nonconforming is greater than , the lot is not accepted. The design of a single sampling plan requires the selection of the sample size and the acceptance number .
If the customer gives NOK (Not OK) at the end of User Acceptance Test, its reasons are examined. If the reason of NOK (Not OK) is a new demand, the process progresses with Design Change Request-Change Request. Design Change Request is a request which causes dramatic changes in infrastructure and effects design step of the product. Change Request is a modification of the existing request or improvement of the product with an additional request.
We can explain all these processes with an example. (In this example, it is assumed that although the steps of analysis review and technical design review are completed, an analysis of needs and design planning different from the customer’s expectations are performed for a reason.)
1. Requirement: A screen is demanded to list the subscribers who are passive or active in a certain time period. In this phase, the demanding person does not know what kind of a screen he/she wants. (Figure.1)
2.Analysis: A screen is designed; the developer is informed that two datetime pickers for giving a time interval, a data grid for listing subscribers and a button for tapping are required on this screen. (Figure.2)
3.1. Development: The developer adds Segment Search textbox in consideration of performance problems created by listing more than one segment in the data grid. (Figure.3)
3.2. Early Customer Review: Figure.3 is shown to the demanding person at the Development phase. It is explained to him/her that there is no need to search segments and they can be statically shown from combobox. Since this issue is determined within Early Customer Review, it is evaluated as an analysis issue.
4. Functional – System Test: After the completion of the development, test engineer decides whether to start test or not by assessing the test approval criterias. If unit test, code inspection and smoke test results are successful functional tests start. The test engineer test positive and negative test scenarios on the screen shown in Figure.4.
5. User Acceptance Test: At User Acceptance Test phase, the demanding person demands a popup screen on which the details can be displayed when it is clicked on the subscriber listen in data grid. This demand is considered within Change Request and the requested change is not done at this phase.
Moreover, at the analysis step, there is an extra status as Suspend in the structure which is designed as Active and Passive. It is realized that Suspend is actually a kind of Passive status and should be considered like that. Thus, the subscribers in the suspend status should be placed in this list. This issue is evaluated as User Acceptance Test since it is determined within this scope.
In the above mentioned example:
• If it was skipped to Step 4 without running Step 3.2, The product would be delivered to the customer with a deficiency in “displaying details of subscribers” and we would have to face a more expensive process : Design Change Request.
• If Production was initiated before Step 5, User Acceptance Test Issue could not be detected and thus a more expensive solution with Production defect would be tried. Meanwhile, we would have to face loss of prestige and customer dissatisfaction.
Statistics of Turkcell within the first three months of 2012 show 7 CR, 1 DCR, and 9 UAT Issues in total 10 sample projects. Accordingly, if User Acceptance Test was not employed, these demands would be returned to testing after the product was sent to production. (Figure.5.1, Figure.5.2)
In accordance with these results; we have detected that Design Change Request and/or Change Requests result from inefficient evaluation and guidance in the phase of needs analysis and that defects in User Acceptance Test rise when test cases are not shared with the analyst.
After the reasons of the ratios in UAT and Early Customer Review were found out and these problems were addressed, we developed three different ways of solution.
Solution 1: After the reception of the demand and the issue of the analysis, the demanding person, the analyst, the developer and the test engineer come together to discuss the analysis. It is expected for everyone to agree upon what is demanded and how it is going to be done. In this way, the possible misunderstandings and insufficient understandings in the next phases would be prevented.
Solution 2: The technical design document prepared by the development is shared with the demanding person, the analyst, and the test engineer and confirmed by everyone.
Solution 3: Before User Acceptance Test, the scope is reconsidered through a discussion of test scenarios with the analyst. If explained in detail: shortly after the completion of product tests and before User Acceptance Test, business analyst tests begin. The aim of Business Analyst tests is to technically examine the requirements. The analyst, the developer and the test engineer participate in Business Analyst tests. Business Analyst test cases and scenarios are documented. Issues in Business Analyst Tests are reported as “Code Issue”.
The fundamental aim of the solutions we have applied are to provide to detect findings on earlier phases. Detecting findings on earlier phases have less cost than detecting them on posterior phases as it is seen in the following tables : (Figure.6.1, Figure.6.2)
Figure.6.2 (Increasing cost of defect correction)
The results obtained in line with these solutions in the last three months of 2012 show defects in 4 CR and 6 UAT out of 10 sample projects. There has not been detected any DCR yet. (Figure.7.1, Figure.7.2)
Actively involving the client in the acceptance test planning and execution process should bring any issues to the surface early. The process gives the client a chance to find problems before the applications are moved into production when finding and fixing defects becomes much more costly. In some instances, the User Acceptance Test Plan may act as a ‘contract’ between the client and the development team, especially when other documentation is insubstantial or missing. 
In sum, we have achieved to decrease customer dissatisfaction to a minimum level and to make customers become familiar with the product.
Asli Karatay Koçak graduated from Istanbul University in 2010 with a computer engineer. She is Senior Software Test Engineer at Turkcell. In this capacity, she is responsible for Information and Communications Technology – Revenue Management Capabilities area of the company. She reports all activities to senior management and Test Architect Committee. She works closely with business analysis team, development team and operational team to ensure SDLC processes are being performed. She also provides certification of Fundamental Test Engineer from ISTQB. She is a member of Turkish Testing Board. She is member of winning team of the competition at Test Istanbul Conference in 24.May.2012 and she has been accepted to present at Conference. She interested in photograph, puzzle, trekking and series.
Lütfiye Yetisen Meliye graduated from Inönü University in 2005 with a degree in economics. She is Senior Software Test Engineer at Turkcell.
In this capacity she is responsible for operational and information systems areas of the Company. She reports all activities to senior management and Test Architect Committee. She works closely with the development team, operational team and business analyst team to ensure SDLC processes are being performed. She also provides certification of Fundamental Test Engineer from ISTQB. She is a member of Turkish Testing Board. She is a member of winning team of the competition at Test Istanbul Conference in 24.May.2012 and she has been accepted to present at the conference. She is interested in books and puzzles.