Regression test selection model: a comparison between ReTSE and pythia

As software systems change and evolve over time regression tests have to be run to validate these changes. Regression testing is an expensive but essential activity in software maintenance. The purpose of this paper is to compare a new regression test selection model called ReTSE with Pythia. The ReTSE model uses decomposition slicing in order to identify the relevant regression tests. Decomposition slicing provides a technique that is capable of identifying the unchanged parts of a system. Pythia is a regression test selection technique based on textual differencing. Both techniques are compare using a Power program taken from Vokolos and Frankl’s paper. The analysis of this comparison has shown promising results in reducing the number of tests to be run after changes are introduced.


Introduction
There are many techniques have been proposed in regression testing racently [1][2][3][4][5][6][7][8][9][10][11]. The most important issue in regression testing is how to minimize the reuse of the existing test cases for the modified program. One of the techniques to tackle this issue is called regression test selection. This technique attempts to reduce the cost of regression testing by selecting appropriate test cases using information from the certified program, the modified program and the existing test suite [12]. Previous regression test selection approaches used an inclusive technique. Inclusive means the techniques select test cases directly from test suite that are needed in regression testing.
Regression test selection techniques can be divided into categories based on the elements used such as control-flow based [1], code entities based [13], textual differencing based [14,15], and program slicing based [16][17][18][19]. Rothermel and Harrold [12] proposed a safe and efficient regression test selection technique based on control-flow graphs (CFG). They developed two main algorithms based on intraprocedural and interprocedural test selection algorithms. The intraprocedural algorithm operates on individual procedures and the interprocedural operates on entire programs or subsystems. In this technique, both the original and modified program will be transformed into a CFG in order to perform a comparison. The comparison algorithm will compare each node in both CFGs. If nodes differ, the algorithm will select tests from an existing test suite that exercised the node. Chen et al. [9] propose a regression test selection technique based on identifying modified code entities such as functions, variables, types, and macros. Test cases that have traversed the modified code entities will be counted in the test suite for the modified program. Vokolos and Frankl [14,15] have developed a tool called Pythia that is used to reduce the cost of regression testing. The Unix-based tool implements an analysis technique that is called textual differencing and works by comparing the source files from the certified and modified programs. The Pythia tool can be used to analyse software systems written in the C programming language.

845
There are a number of regression test selection techniques based on program slicing techniques. Binkley [16] conducted a survey about the application of program slicing to regression testing. He divided the technique into three groups of program slicing that are used in regression testing. The first group uses dynamic slicing, the second group uses program dependent graphs (PDG), and the third group is based on Weiser's data flow definition of slicing [20]. Agrawal et al. [18] have proposed three algorithms to be used in their technique called incremental regression testing. The algorithms are an execution slice, a dynamic slice, and a relevant slice. The execution slice of the program with respect to a test case is referred to as the set of statements executed under that test case. The dynamic program slice with respect to the output variables gives the statements that are not only executed but also have an effect on the program output under that test case. The relevant slice with respect to the program output for a test case is referred to the set of statements that, if modified, may alter the program output for the given test case.
Gupta et al. [19] have developed a data flow based regression testing technique that uses slicing algorithms to explicitly determine the affected definition-use associations made by a program change. The technique uses two slicing algorithms to detect directly and indirectly affected def-use associations. The first algorithm works backward from the changed statement to its definitions. The second algorithm is a forward walk from the same point as the first algorithm. The forward algorithm detects uses, and subsequent definitions and uses, which are affected by a definition that is changed at that point. Vedpal and Chaulan [21] have proposed a regression test selection technique based on identification of affected paths, affected functions and dynamic slicing which can be used to reduce the number of test cases for regression testing. This technique has two main processes. The first process is the construction of the OPDG (Object Oriented Program Dependency Graphs) for the modified program. The second process is a selection of test cases based on an algorithm that involved dynamic slicing. Qu et al. [22] have proposed a configuration selection approach for regression testing. Their technique addresses the solution to the redundancy issue in retest-all approach for configurable systems. The technique has employed an existing static slicing tool called CodeSurfer for computing the configurable option impact and the code change impact.
Xing et al. [23] have developed a new regression test selection technique based on the program dependence graphs of the original program and its modified version. The technique has embedded slicing technique in their two main steps. Comparing with previous techniques, they claimed their technique can eliminate some unnecessary tests to rerun. All these slicing based RTS techniques are classified as inclusion techniques which select test cases from the test suite that are needed in regression testing. The idea of the regression test selection by exclusion was proposed by Gallagher et al. [24]. Ngah et al. [25] have developed a new regression test selection by exclusion using decomposition slicing called ReTSE. Exclusion technique omits test cases from test suite that are not needed in regression testing. This paper presents the comparison between ReTSE model and Pythia technique.

ReTSE Model
The ReTSE model [25] is a four phase model that has three inputs and three outputs. Phase 1 has two main steps which are Pretty Print Step and Slicing Step. This phase will produce decomposition slices for both Certified Program (C) and Modified Program (M). In Phase 2, the decomposition slices will be compared in order to produce a set of Similar Decomposition Slices (S), Different Decomposition Slices (D), Deleted Decomposition Slices (L) and New Decomposition Slices (N). In Phase 3, any test cases that not related to the D will be excluded from test suite. Finally in the Phase 4, the model will identify any redundant test cases that may exist in regression tests produced in Phase 3. More details explaination about the ReTSE model can be founded in [25].

Results and Analysis
Vokolos and Frankl [14] have used Power program to illustrate the Pythia, a textual differencing technique. The program aims to raise a floating point number to an integer power, using Dijkstra's algorithm. The program consists of two files: main.c and power.c. Each file contains one function. The old version of the power program (power.c) is shown in Figure 1. Because the ReTSE model only concentrates on intraprocedural, it is assumed that the changes only occur in the power.c. The new version of power function (power-v1.c) is shown in Figure 2. The old version is called Certified Program (C) and the new version is called Modified Program (M) in the ReTSE model.  Table 1. Test case TC2 in Table 1 catches an error in the certified program. The THi of each TCi for C (power.c program) is shown in Table 2. In their model, THi is called a basic block execution trace which is an execution trace of a test case based on a basic block concept. A basic block is a sequence of consecutive statements with the property that control enters at the beginning statements and may leave only at the very last statement [14]. However, the THi in Table 2 is slightly different from their paper in order to consider a statement based used in the ReTSE model. The X symbol in Table 2 shows that the statement is executed for that TCi. The -symbol is for statement not executed.

Phase 1: Program Analysis
The Original Certified Program and the Original Modified Program of the power program are assumed to have gone through the Pretty Print Step. The outputs of this step are a Certified Program (C) and Modified Program (M) as shown in Figure 2 and Figure 3 respectively. The M has two additional new statements at S10' and S11' in order to tackle the problem raised in the Certified Program (C).
In the Slicing Step, both C and M are decomposed into decomposition slices corresponding to their variables in the programs. Therefore, both programs have five decomposition slices corresponding to five variables which are x, recip, n, sgn and y. Decomposition slices for variables sgn and y are similar as shown in

Phase 2: Comparison
In the first part of the Comparison Phase, the decomposition slices in the DS-C are compared to the decomposition slices in the DS-M using the diff tool. There is no output produced from the diff tool for the comparison between DS-Cx and DS-Mx. Similar result was achieved in the comparison between DS-Crecip and DS-Mrecip. Therefore, these decomposition slices are included in a set of pairs of Similar Decomposition Slice (S). An output is produced from the diff tool for the comparison between DS-Cn and DS-Mn as shown in Figure 3. Therefore, both decomposition slices are included in a set of pairs of Difference Decomposition Slice (D). The DS-Csgn and DS-Msgn as shown in Figure 4 are also included in the D because the output is produced from the diff tool for this comparison. The DS-Cy and DS-My in Figure 4 are also included in the D because the diff tool also produces an output from this comparison. The output summary of the first part of the Comparison Phase is shown in Table 3.  (RCSsgn). Therefore, S9 from DS-Csgn is included in the CSsgn and statements S10' and S11' from DS-Msgn are included in the RCSsgn. Statement S9' is also included in the the RCSsgn because it is located at the same branch of statement S10'. The same happens to the comparison between DS-Cy and DS-My as shown in Figure 4 where statement S9 from DS-Cy is included in the CS y and statements S9', S10' and S11' from DS-My are included in the RCSy.
In the comparison between DS-Cn and DS-Mn as shown in Figure 3, line 8 ([L8]) is not a statement but a close curly bracket (}). Therefore, the statement immediately after that symbol will be included in the set of Change Statement for variable n (CSn). Any statement in the range of lines 9 ([L9]) to 12 ([L12]) from DSMn is included in the set of Relevant Change Statement for variable n (RCSn). Therefore, statement S10 is included in the CSn and statements S8' and S10' are included in the RCSn. Then the CS is produced from the union of CSsgn, CSy and CSn where the RCS is produced from the union of RCSsgn, RCSy and RCSn. A summary of the second part of the Comparison Phase is given below: CS = CSsgn U CSy U CSn = {S9} U {S9} U {S10}= {S9, S10} RCS= RCSsgn U RCSy U RCSn = {S9', S10', S11'} U {S9', S10', S11'} U {S8', S10'} = {S8', S9', S10', S11'}

Phase 3: Exclusion
There are five test cases (TCi) that were used by Vokolos and Frankl for Certified Program (C) of power.c program as shown in Table 1. Their THi is shown in Table 2. Any TCi where the CS is not subset of THi, will be included in the set of Excluded Test (ET). In this case, the CS which includes statements S9 and S10, is not subset of TH3, TH4 and TH5. Therefore, TC3, TC4, and TC5 will be included in the set of ET. The remaining test cases in Test Suite are included in the set of Regression Tests (RT). Therefore, TC1 and TC2 are included in the RT. A summary of the Exclusion Phase is given below:

Conclusion
The Pythia selected two test cases from an existing test suite to test a new version of the power program [10]. The same program has been applied to the ReTSE model. In the Exclusion Phase, the ReTSE model has selected the same two test cases as the Pythia technique. Moreover, after Optimisation Phase, the ReTSE model has selected only one test case to test a new version of the power program. This is because the model has identified that both test cases (produced in the Exclusion Phase) are redundant at the same coverage of a new version of the program.