Automatic test-case generation for system testing

ISSN 2354-0575 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Journal of Science and Technology 61 AUTOMATIC TEST-CASE GENERATION FOR SYSTEM TESTING Minh-Hue Chu*, Trinh Thi Nhi, Do Thi Thu Trang, Ngo Thanh Huyen, Tran Do Thu Ha Hung Yen University of Technology and Education * Corresponding author: huectm@gmail.com Received: 15/06/2020 Revised: 20/08/2020 Accepted for publication: 25/09/2020 Abstract: In this paper, we propose a method to generate test cases automatically from a detailed use case specification for system-level functional testing. Our method consists of two parts: (1) Propose UCML (Use Case Modeling Language) to describe the use cases in more detail; (2) Automatic test scenarios generation algorithms and corresponding test cases from detailed use case specification model by UCML. Test scenarios are generated based on the branch coverage criteria. Test cases are generated based on the test scenarios coverage criteria and subclauses coverage criteria of the constraint conditions in each test scenario. Each test case provides the following details: Test execution precondition, input data, expected output, and test execution steps are the basis for writing self-test automated execution scripts. Keywords: automatic test case generation, functional testing, system testing, testing based on use case, UCML. 1. Introduction Software testing is an essential activity in the software development process to ensure software quality. Nowadays, the size and complexity of software are increasing. This makes to raise the cost of time and manpower for testing activity. Proposal and development of automated test techniques becomes imperative. Software testing process can be divided into three main phases: test cases design, testing execution, and test evaluation. In particular, the automated test cases design is a phase that commonly faces difficulties and challenges. In functional testing, the test cases design is usually done manually based on the software requirement specification document. Normally, functional requirements are often described by use case diagrams and use case specification documents that are described by natural language according to a template such as in [1]. So, the automatic test case generation from use case specification is still a challenge for the research community. The test method based on the use cases is described by Jacobson in [2]. Firstly, Test scenarios are identified from each use case specification. Nextly, a set of test cases is identified from each test scenario to realize the test scenario. There was much research focused on specifying the use case to generate test scenarios and test cases automatically. Such as in [3], the authors used the markup language structure to specify the use case scenarios. This specification was then passed on to the activity diagram. Finally, the test scenarios were generated base on the basis path coverage criteria of the activity diagram. The authors in [4] proposed a contract structure to specify the pre-condition and post-condition of each use case execution step. Sequence diagrams were used to describe use cases. From these specifications, the test scenarios can be generated automatically. The authors in [5] defined a Requirements Specification Language (RSL) that allows specifying use case scenarios. This specification was then transformed automatically to test cases specified in the Test Specification Language (TSL). The main limitation of this approach is that use case scenarios must be manually specified in RSL from requirement documents expressed in natural language. The authors in [6] used a finite state machine to specify the system behavior corresponding to use cases. Each finite state machine is browed to ISSN 2354-0575 Journal of Science and Technology62 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 generate test cases. The drawback of this approach is that we must manually define the states from the requirement specification. The authors in [7] proposed the UMTG tool. This research used a restricted natural language with a set of keywords to specify use case scenarios. These specifications were then used to generate test scenarios by the Natural Language Processing (NLP) technical. Generated test scenarios achieve the execution path coverage criteria. OCL (Object Constraint Language) constraints are solved to generate Test data. Test data is generated by solving OCL constraints. Another approach as in [8], [9], [10], [11], [12], researches used UML activity diagrams to specify use case scenarios. Test cases are then generated from these UML activity diagrams. The main advantage of the activity diagram is that it is simple, easy to use, and easy to understand for the logic flows of the system. However, the above approach mainly focuses on guiding the method of determining different paths on the activity diagram based on the excution path cover criteria. Test data is little mentioned. The coverage criterion consists of many types [13]: statement, branch, sub-conditions clause, and path coverage criterion. In there, the branch coverage criterion ensures the statement coverage criterion, and the path coverage criterion ensures the branch coverage criterion. However, the path coverage criterion is the highest coverage criterion. The combination of this criterion with the sub- condition clause coverage criterion will ensure the best possible test cases. However, the path coverage criterion is not usually achieved with loops. In this paper, we propose a formal specification language named UCML (Use Case Modelling Language) to capture use case descriptions with automatic test case generation purpose. We built algorithms to generate test scenarios and test cases from UCML specifications. UCML language is built base on UML activity diagram and adds the concept Contract. Each contract is attached to the action or flow to specify the details for an actions or a flow. It allows extracting more detail information to generate test scripts and test cases. We propose two algorithms: Algorithm 1 allows generating test scenarios with a set of contracts on each scenario based on the excution path coverage criteria as presented in [11]; Algorithm 2 allows determining the sub-clauses that satisfy the conditional clause. These clauses allows determinating of test cases from test scenarios that these test cases cover all test scenarios and all subclauses in conditional clauses on each scenario. 2. Use case Specification Language UCML This section focuses on presenting use case formal specification language UCML for the purpose of automatically generating test cases. We use the example of Withdraw use case in ATM system [9] to illustrate the proposed method. Figure 1. UCML FrameWork Figure 1 shows the main activities of software engineer in the UCML framework to automatically generate test cases from use cases. The UCML framework consists of two main components: (1) UCML Editor supports specifing use cases in the UCML formal specification language in the form of graphical models. To build the UCML modeling language and an editor for the language we use Eclipse’s EMF and GMF projects [14]; (2) UCML Generator allows automatically generating test scenarios and test cases from UCML models. This module is built by the model to text (M2T) technique (Gronback, 2009) and based on the generation algorithms described in Section 3.2. 2.1. Example Use case Withdraw in the ATM system allows system users to make withdrawals from the system’s ATM account [9]. The detailed use case specification is described in Table 1. A use case specification includes pre and post- condition, trigger event basic Flow, and alternate flows. Scenario tests and test cases are generated based on the use case, conditions of flows, pre- ISSN 2354-0575 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Journal of Science and Technology 63 condition, and post-condition information in the use case specification. UCML specification language allows describing a use case in a visual graph model. We build then algorithms in the M2T model transformation language to convert this formal specification to test scenarios and test cases. Figure 2 depicts the Withdraw use case in the UCML language. Table 1. Use case description Withdraw [9] Use-case: Withdraw Primary Actor: User 1. Pre-condition Insert Card Use Case was success 2. Post-condition If the use-case was successful, the systemupdates the balance, dispenses the cash, prints a receipt for the user. If not, the system displays an error message. 3. TriggerUser Click “Menu Withdraw” Basic flow 1 A: The actor enters the amount she/he wants to withdraw 2 S:the system checks whetherthe value entered is valid 3 S: the system will retrieve the balance of the user then checkif it is sufficient to withdraw 4 S: the system updating the balance, dispensing the cash and finally printing a receipt for the user Alternate flows 2a the actor enters an invalid amount 1. S: the system displays an error message “Amount is invalid, Enter a valid amount” 3a the balance isn’t sufficient 1. S: the system checks whether the user has permission to overdraft. If has go next, if don’t has go to 3b 2. S. the system compares the amount required with the maximum allowed limit for overdraft, if the amount needed is within limit, go to 5, else go next 3. S. the system displays an error message “Amount is beyond Limit” 3b the user has no overdraft permission 1. S: The system displays an error message “you don’t has permission withdraw over amount in your account” 2.2. Abstract Syntax of UCML The UCML language is proposed on the basis of the UML activity diagram extension. We add the concept of Contract to describe the constraints for use case execution steps. Concepts in the UCML language include: - InitialNode, FinalNode are the initial node and final node of a use case, respectively. - ActorAction, SystemAction are an actor action and a system action, respectively. - DecisionNode is a decision node that has many different outgoing flows depending on the conditions. - Flow is a directional flow between actions; - Contract describes details constrains of each action or control flow; - Particularly, Contract has the following structure: - Contract Name - Pre Condition - Post Condition - HasContract is the link between an action or flow with the Contract attached to it. - Name is name of Contract. - that must be fulfilled in order to the action to occur, or to follow the flow. - Condition of Post is logic expression describing expected outputs. - Condition of Pre is logic expression describing the constraints on the input data. The abstract syntax of UCML is built by a metamodel as in Fingure 2. The specification shown in Fingure 3 conforms to this metamodel. The language concepts are represented by graphic notations. 3. Generating test case from UCML Specification In this section, we present the main idea to generate test cases from the UCML use case specification. Next, we present algorithms to generate test case scenarios based on the execution path and sub-clause coverage criteria. Figure 2. Metamodel of UCML ISSN 2354-0575 Journal of Science and Technology64 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Fingure 3. Specifying use case Withdraw by UCML 3.1. Main Idea In the use case driven test method in [2], we will identify the test cases from the use case specification according to the following steps. - Step 1: We define the use case scenarios. For each use case scenario is a test scenario. - Step 2: For each test scenario, we identify different test cases with inputs satisfying the constraints on the scenario execution path. We denote D as a graph corresponding to a use case specification in UCML. From specification D, execution paths can be determined based on the criteria of covering all execution paths and the set of contracts corresponding to each execution path. Each execution path is a test scenario. Test execution steps of a test case corresponding to actor actions on the execution path. Test cases are identified by traversing a set of test scenarios. Each scenario identifies different test cases based on the satisfaction of subclauses in the conditional clause declared in the Pre of the Contract. The expected outputs are taken from the Post declaration of the set of contracts. The pre-condition of a test case is taken from the Pre declaration in the Contract attached to the InitialNode node. Test scenarios are generated based on the execution path coverage criteria. Tests are generated base on the scenarios coverage criteria and sub-conditional clauses coverage criteria. 3.2. Automatic Generation Method In this section, We propose two algorithms for generating scenarios and test cases from graph D: Algorithm 1 identifies different execution paths with the Contract set on each execution path. Corresponding to each execution path, we identify a test scenario. Algorithm 2 determines different value tuples of sub-clauses satisfying the conditional expression. The purpose of Algorithm 2 is to define test cases for a test scenario based on the sub-clauses coverage criteria. Graph D corresponds to a UCML specification. Its structure is similar to an activity diagram in UML. Algorithm 1 below identifies the different execution paths in graph D and sets of constraints on execution paths. Determined execution paths reach to cover all activity paths. To define execution paths in D, we use a depth-first graph traverse algorithm from the start node to the end node. At each action, the path will continue to be explored through the next actions with the flow out from the action. Next action is visited if the action has not been passed, or the action only constitutes a sequence of actions that occur twice in the path. To determine whether a flow can go forward, we construct the NextStepCondition (p, a, t) function, where p is the set of actions that went on the path from the initial node to the action a, t is the coming outflow from a. The function returns True if action a follows flow t, it returns False if action cannot continue to flow t. Algorithm 1: Defining execution paths of a use case and set Contracts along the paths. Input UCML model D=(A, T, Fout , Fin, C ,a0 , aL). Output P: Set of execution paths in D. V: Set of Contracts of each use case scenarios. (D, P, V are global variables) The variable x is the node currently under ISSN 2354-0575 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Journal of Science and Technology 65 consideration (x ! A), p is the path (with order) from the initial node to a node being before node x, v set of contracts (with order) on the path p. Start 1: Procedure AllPath(x, p, v) 2: Begin 3: p = p j {x}; 4: If 7 ConA(x) and (x ! a 0 ) then 5: v = v j {ConA(x)}; 6: If x = a L then 7: P = P j {p}; V = V j {v}; Return; 8: EndIf 9: ForEach y in {h ! A | 7 t ! T: (x, t) ! Fout & (t, h) ! F in } do 10: If NextStepCondition (p, x, t) then 11: p’ = p; v’ = v; 12: If 7 ConF(t) then v’ = v’ j {ConF(t)}; 13: AllPath(xn, p’, v’); 14: EndIf 15: EndFor 16: End. 17: P = Q ; V = Q ; 18: p = Q ; v = Q ; 19: AllPath(a 0 , p, v); Stop Algorithm 1 determines P containing all the different execution paths. These paths satisfy the execution paths coverage criteria in graph D. Algorithm 1 also determinis V containing all set of Contrains on each path in P. With a path, the algorithm determines actor actions to identify test execution steps. For each execution path, by traversing the set of contracts we get the bound conditional clauses along the path. Each conditional clause will be solved in order to obtain different value tuples of satisfied sub-clauses. For each matching tuple, we define a test case. To reduce the number of test cases with duplicate inputs, the algorithm uses a global variable that stores the flags of all contracts. Flag of a Contract will be set when a Contract is met in a test scenario. If a Contract has not been set, then all sets of different input values will be taken when analyzing the conditional clause in the Pre declaration of Contract to identify different test cases corresponding to the scenario test. Contrary, when the flag value of Contract has been set, the algorithm takes only a set of input values to define a test case. For example, there are two execution paths p1 and p2, the set of contracts corresponding to two execution paths p1 and p2 that contain Contract C_2 with the conditional clause declared in Pre as follows: “(A <0) or (B <0 )”. When calculating sub-conditional statements that they satisfy C_2, we identify 3 different sets of values satisfy: “A <0”, “B <0”, “A <0 and B <0”. When traversing p1 to generate test cases, if the flag of C_2 does not set, corresponding to C_2 will define 3 different test cases with input data satisfying C_2. The flag of C_2 will be then set. When traversing p2 to generate test cases, with the flag of C_2 that is set, corresponding to C_2 will define a test case with any input data of 3 predefined values. Algorithm 2: Solving the conditional clause Input E: the post-conditional expression OutPut R: stacks save expresions satisfying E. Start 1: R = Q ; Stack r = Q ; R = Rj{r}; 2: Foreach item in E do 3: Case item of 4: “and”: 5: Foreach r in R do 6: String st1 = r.Pop() ; String st2 = r.pop(); 7: r.Push(st2 + “ and “+ st1 ); 8: EndFor 9: “or”: 10: Foreach r in R do 11: String S1 = r.Pop(); String S2 = r.Pop(); rn = r; 12: r.Push(s1); rn.Push(s2); R = Rj{rn}; 14: EndFor 15: “xor”: 16 Foreach r in R do 17: String s1 = r.Pop(); String s2 = r.Pop(); rn = r; 18: rn.Push(s2 + “ and not “ + s1) ; 19: r.Push(s1 + “ and not “ + s2) ; 20: EndFor 21: Else // operand 22: Foreach r in R do 23: r.Push(item); 24: EndFor 25: EndCase 26: EndFor Stop. Algorithm 2 above solves the conditional clause. The input of the algorithm is the stack that stores the conditional expression converted to a suffix. Particularly, each element in the stack is an operator or logical operands: “or”, “xor”. For example, conditional expression: “(A and B) or C and (D or E)” are converted to suffix expression: “A”, “B”, “and”, “C”, “D” , “E”, “or”, “and”, “or” and stored in stack E. From E, algorithm 2 removes the duplicate values. We will get the following results: “A and B “,” C and D “,” C and E “. These values are stored in stack R. Corresponding to each element in R is a stack that stores the sub- expression satisfying the conditional expression in E. Combining all of the value in R by the operation “and” we obtain an additional satisfied value such as “A and B and C and D and E”. ISSN 2354-0575 Journal of Science and Technology66 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 4. Experiment Results and Discussion Test scenarios of the Withdraw use case are generated as shown in Table 2. Actions are numbered based on the use case specification as described in Table 1. Test cases are determined from the respective paths above of the Withdraw use case as shown in Table 3. In this section, we compare our approaches with some other approaches. The authors in [3] have only identified different test scenarios, and have not identified specific test cases for each scenario. With the method proposed in [4], the us