Chapter 5

REGRESSION TEST

Regression testing is the process of validating modified parts of the software, hoping that the modification does not introduce errors that were not exists before the modification. An important different between regression testing and development testing is that during the process of the regression test, a set of established tests can be reused. Regression test cases is a set of test case aiming at testing the core function or procedure of the software; core function means where procedure call with known input is made and the output of the function will be compare with output the developer expect. The set of test cases will then be reused for later development as incremental process, as development gets progress the size of the regression test will also be increase. In other word, the regression test result in greater test coverage, having greater coverage will eliminate the amount of errors/bugs in the end product, which is the ultimate goal of having test cases. Another advantage of the regression test is that all the tests are automated; all the test cases that gather before will be run all together without manually inputting date and comparing the output. These can significantly increase the efficiency of the testing process in developing a software product.

The simplest regression test strategy is retest all, when regression test is started it re-run all the test cases that the test suite have, having this approach comes in a huge cost in terms of time. In a commercial software source code can reach up to several millions lines of source code and several hundred thousands of procedure, having a set of test testing all of these procedures each time when there is a modification of the software may require unexpected amount of time. An alternative approach is to have regression test selection, unlike retest all use all the test cases where as this approach will use some selection techniques to select a set of test case to be used. Using this regression test technique, time can be save but having this approach will also have some negative aspects, for example, the cost of regression test technique can be high if sophisticate technique were used, where as low cost with a low quality technique, and the selection may discard test cases that could expose errors, in other word reducing the fault detection effectiveness.

Before discussing the techniques for regression test selection, we should notice that during the development process specification of the software may be change, if the specification has changed it is necessary to remove those test cases that were reflecting the old specification from the test suite before perform regression test selection. This is important because if we cannot correctly determine the test case is out of date or not, we cannot correctly determine the correctness of the test cases.

Regression Test Selection Technique

Before discussing some different types of techniques, it is helpful to have some notation defined first:

Let P be a procedure or program; let P’ be a modified version of P; and let T be a test suite for P. T’ is a subset of T, and set of test cases to execute on P’

These techniques discuss below has been discuss in [2]

· Minimization Techniques: This technique involve selecting a minimal set of test cases from T which cover the modified section of P, P'. In a sample scene, it require T' to at least one test case to cover those area where modification had made.

· Dataflow Techniques: This technique select test case that exercise data interaction that have been affected by the modifications.

· Safe Techniques: Most regression test selection is considered to be not safe; not safe means that the selection fail to select those test cases that would reveal a error in the modified program. With safe regression test selection techniques, it is guarantee that the selection T' (subset of T) would contain all the test case in the original test suite T that able to reveal errors in P'.

· Random Techniques: As the name shown, test cases is randomly select from T.

· Retest All Techniques: The Retest All techniques simply re-use all the test case against P'.

Regression Test on OpenACS Calendar

The regression test selection techniques that have chosen is Retest All, the main disadvantage of this technique is that if it run against large software having several thousands functions, the time it needs to complete the test is enoumes and the computing power it needs is very expensive. But in terms of coverage Retest All provide the maximum coverage out of those techniques discuss in the previous section; again, having better coverage means the tested software will more likely to discover errors before the software go into production. With OpenACS Calendar, it is a relative small program with less than 50 functions and less than 3000 lines of source code, therefore it is crucial to use the Retest All technique in order to achieve the best possible result; to find errors as many as possible during the test phase. As we have discuss in the beginning of this chapter regression test should be automated, one of the tools provided in OpenACS called aa-test [5], it is a tool kit which allow developer to register test cases (this require developer to program how to test a piece of code and the expected result, the expected result will be compare against the actual result) with the system, this tool will build up all the test cases that registered, then developer have choice to run all test case in once or select individual test case. Result will be available to view on the browser and result statistics will also be logged. We have chose to use this automated test tool as out regression test tool. A sample regression test case is shown below in Figure 10. A complete regression test source code on OpenACS Calendar is attached in the Appendix C.

Figure 1 the input arguments and return of function cal_item_create

Figure 2. regression test case for testing function cal_item_create