Regression Testing.pdf

05-09-2024 REGRESSION TESTING Dr. Madhusmita Sahu Assistant Professor Department of Computer Science and Information Technology Topics to be discussed Regression Re...

05-09-2024 REGRESSION TESTING Dr. Madhusmita Sahu Assistant Professor Department of Computer Science and Information Technology Topics to be discussed Regression Regression Regression Testing Testing Testing Types Techniques Selective Retest Technique Regression Test Prioritization 1  05-09-2024 Regression Testing ⬢ After a piece of code has been successfully tested, changes to it later or may be needed for various reasons. ⬢ Regression testing is the practice of running an old test suite after each change to the system or after each bug fix to ensure that no new bug has been introduced due to the change or the bug fix. Regression Testing ⬢ However, if only a few statements are changed, ⬡ then the entire test suite need not be run— ⬡ only those test cases that test the functions and are likely to be affected by the change need to be run. 2  05-09-2024 Regression Testing ⬢ A system under test ( SUT) is said to regress if ⬡ a modified component fails, or ⬡ a new component, when used with unchanged components, causes failures in the unchanged components by generating side-effects or feature interactions. Regression Testing ⬢ The purpose of regression testing is to ensure that bug-fixes and new functionalities introduced in a new version of the software do not adversely affect the correct functionality inherited from the previous version. 3  05-09-2024 Regression Testing ⬢ Regression testing is the selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements. ⬡ IEEE software glossary Regression Testing ⬢ Regression testing can be defined as the software maintenance task performed on a modified program to instill confidence that changes are correct and have not adversely affected the unchanged portions of the program. 4  05-09-2024 Regression Testing Baseline version The version of a component (system) that has passed a test suite. Delta version A changed version that has not passed a regression test. Delta build An executable configuration of the SUT that contains all the delta and baseline components. Regression Testing Types Side-Effect regression/St ability regression Bug-Fix It involves retesting a regression substantial part of the product. 01 02 This testing is The goal is to prove performed after a that the change has bug has been no detrimental effect reported and fixed. on something that Its goal is to repeat was earlier in order. the test cases that It tests the overall expose the problem integrity of the in the first place. program, not the success of software fixes. 5  05-09-2024 Notations used in Regression Testing Problem Notation Description P denotes a program or procedure P’ denotes a modified version of P S denotes the specification for program P S’ denotes the specification for program P’ P(i) refer to the output of P on input i P’(i) refer to the output of P’ on input i T={t1, t2, …, tn} denotes a test suite or test set for P Regression Testing Problem ⬢ Given a program P, its modified version P ‘, and a test set T that was used earlier to test P; ⬡ find a way to utilize T to gain sufficient confidence in the correctness of P ‘. 6  05-09-2024 Regression Testing Techniques Regression test 01 selection technique Test case prioritization 02 technique Test suite reduction 03 technique Selective Retest Technique ⬢ Software maintenance includes more than 60% of development costs. ⬢ Selective retest techniques attempt to reduce the cost of testing by identifying the portions of P ‘ that must be exercised by the regression test suite. 7  05-09-2024 Selective Retest Technique ⬢ Selective retesting is distinctly different from a retest-all approach that always executes every test in an existing regression test suite ( RTS). ⬢ Thus, the objective of selective retest technique is cost reduction. ⬢ It is the process of selecting a subset of the regression test suite that tests the changes. Features Of Selective Retest Techniques It minimizes the resources required to regression test a new version It is achieved by minimizing the number of test cases applied to the new version It is needed because a regression test suite grows with each version, resulting in broken, obsolete, uncontrollable, redundant test cases It analyses the relationship between the test cases and the software elements they cover It uses the information about changes to select test cases 8  05-09-2024 Steps in Selective Retest Technique Steps in Selective Retest Technique ⬢ Select T ‘ subset of T, a set of test cases to execute on P ‘. (Regression test selection problem) ⬢ Test P ‘ with T ‘, establishing correctness of P ‘ with respect to T ‘. (Test suite execution problem) 9  05-09-2024 Steps in Selective Retest Technique ⬢ If necessary, create T ‘’, a set of new functional or structural test cases for P ‘. (Coverage identification problem) ⬢ Test P ‘ with T ‘’, establishing correctness of P ‘ with respect to T ‘’. (Test suite execution problem) Steps in Selective Retest Technique ⬢ Create T ‘’’, a new test suite and test execution profile for P ‘, from T, T ‘, and T ‘’. (Test suite maintenance problem) 10  05-09-2024 Regression test selection problem ⬢ The problem is to select a subset T ‘ of T with which P ‘ will be tested. Coverage identification problem ⬢ The problem is to identify portions of P ‘ or its specifications that requires additional testing. 11  05-09-2024 Test suite execution problem ⬢ The problem is to execute test suites efficiently and checking test results for correctness. Test suite maintenance problem ⬢ The problem is to update and store test information. 12  05-09-2024 Strategy for Test Case Selection ⬢ Effective testing requires strategies to trade- off between the two opposing needs of amplifying testing thoroughness on one side (for which a high number of test cases would be desirable) and reducing time and costs on the other (for which the fewer the test cases, the better). Strategy for Test Case Selection ⬢ Given that test resources are limited, how the test cases are selected is of crucial importance. ⬢ A decision procedure for selecting the test cases is provided by a test criterion. 13  05-09-2024 Selection Criteria Based on Code Fault-revealing test cases Modification- revealing test cases Modification- traversing test cases Fault-revealing test cases ⬢ A test case t detects a fault in P‘ if it causes P’ to fail. ⬢ Hence t is fault-revealing for P’. 14  05-09-2024 Modification-revealing test cases ⬢ A test case t is modification-revealing for P and P’ if and only if it causes the outputs of P and P’ to differ. Modification-traversing test cases ⬢ A test case t is modification-traversing if and only if it executes new or modified code in P’. 15  05-09-2024 Regression Test Selection Techniques Minimization techniques Retest-all technique 01 Dataflow Ad 05 02 techniques hoc/Random techniques Safe 04 03 techniques Minimization Techniques ⬢ Minimization-based regression test selection techniques attempt to select minimal sets of test cases from T that yield coverage of modified or affected portions of P. 16  05-09-2024 Minimization Techniques ⬢ For example, a technique may use systems of linear equations to express relationships between test cases and basic blocks (single- entry, single-exit, sequences of statements in a procedure). Minimization Techniques ⬢ The technique may use a 0-1 integer programming algorithm to identify a subset T ‘ of T that ensures that every segment that is statically reachable from a modified segment is exercised by at least one test case in T ‘ that exercises the modified segment. 17  05-09-2024 Data flow Techniques ⬢ Dataflow coverage-based regression test selection techniques select test cases that exercise data interactions that have been affected by modifications. ⬢ For example, every definition-use pair that is deleted from P, new in P ‘, or modified for P ‘, should be tested. Data flow Techniques ⬢ The technique selects every test case in T that, when executed on P, exercised, deleted, or modified definition-use pairs, or executed a statement containing a modified predicate. 18  05-09-2024 Safe Techniques ⬢ Techniques that are not safe can fail to select a test case that would have revealed a fault in the modified program. Safe Techniques ⬢ In contrast, when an explicit set of safety conditions can be satisfied, safe regression test selection techniques guarantee that the selected subset T ‘ contains all test cases in the original test suite T that can reveal faults in P ‘. 19  05-09-2024 Safe Techniques ⬢ For example, some techniques may use control-flow graph representations of P and P ‘ and test execution profiles gathered on P, to select every test case in T that, when executed on P, exercised at least one statement that has been deleted from P, or that, when executed on P ‘, will exercise at least one statement that is new or modified in P ‘ (when P is executed, a statement that does not exist in P cannot be exercised). Ad hoc/Random Techniques ⬢ When time constraints prohibit the use of a retest-all approach, but no test selection tool is available, developers often select test cases based on ‘intuitions’ or loose associations of test cases with functionality. ⬢ Another simple approach is to randomly select a predetermined number of test cases from T. 20  05-09-2024 Retest-all Technique ⬢ The retest-all technique simply reuses all existing test cases. ⬢ To test P ‘, the technique effectively selects all test cases in T. Evaluating Regression Test Selection Techniques Inclusiveness Precision Efficiency 21  05-09-2024 Inclusiveness ⬢ It measures the extent to which M chooses modification revealing test from T for inclusion in T ‘, ⬡ where M is a regression test selection technique. Inclusiveness ⬢ Suppose, T contains n tests that are modification-revealing for P and P ‘, and suppose M selects m of these tests. ⬢ The inclusiveness of M relative to P and P ‘ and T is: 1. INCL(M ) = (100 x (m/n)%, if n ≠ 0 2. INCL(M )% = 100%, if n = 0 22  05-09-2024 Inclusiveness ⬢ If for all P, P ‘, and T, M is 100% inclusive relative to P, P ‘, and T, then M is safe. Inclusiveness Example ⬢ If T contains 100 tests of which 20 are modification-revealing for P and P ‘, and M selects 4 of these 20 tests, ⬡ then M is 20% inclusive relative to P, P ‘, and T. ⬢ If T contains no modification-revealing tests, ⬡ then every test selection technique is 100% inclusive relative to P, P ‘, and T. 23  05-09-2024 Precision ⬢ It measures the extent to which M omits tests that are non-modification revealing. ⬢ Suppose T contains n tests that are non- modification revealing for P and P ‘, and suppose M omits m of these tests. Precision ⬢ The precision of M relative to P, P ‘, and T is given by ⬡ Precision = 100 x (m/n) % if n ≠ 0 ⬡ Precision = 100 % if n = 0 ⬢ If T contains no non-modification-revealing tests, ⬡ then every test selection technique is 100% precise relative to P, P ‘, and T. 24  05-09-2024 Precision Example ⬢ if T contains 50 tests of which 22 are non- modification-revealing for P and P ‘, and M omits 11 of these 22 tests, ⬡ then M is 50% precise relative to P, P ‘, and T. Efficiency ⬢ The efficiency of regression test selection techniques is measured in terms of their space and time requirements. ⬢ Where time is concerned, a test selection technique is more economical than the retest- all technique, if the cost of selecting T ‘ is less than the cost of running the tests in T–T ‘. 25  05-09-2024 Efficiency ⬢ Space efficiency primarily depends on the test history and program analysis information a technique must store. ⬢ Thus, both space and time efficiency depends on the size of the test suite that a technique selects, and on the computational cost of that technique. Regression Test Prioritization ⬢ Regression test prioritization attempts to reorder a regression test suite so that those tests with the highest priority, according to some established criterion, are executed earlier in the regression testing process than those with a lower priority. 26  05-09-2024 Regression Test Prioritization ⬢ By prioritizing the execution of a regression test suite, these methods reveal important defects in a software system earlier in the regression testing process. Regression Test Prioritization ⬢ The intention behind prioritizing the execution of a regression test suite is to ensure that the defects normally introduced by competent programmers are discovered earlier in the testing process. 27  05-09-2024 Steps in Regression Test Prioritization ⬢ Select T ‘ from T, a set of test to execute on P ‘. ⬢ Produce T ‘p , a permutation of T ‘, such that T ‘p will have a better rate of fault detection than T ‘. ⬢ Test P ‘ with T ‘p in order to establish the correctness of P ‘ with respect to T ‘p. Steps in Regression Test Prioritization ⬢ If necessary, create T ‘’, a set of new functional or structural tests for P ‘’. ⬢ Test P ‘ with T ‘’ in order to establish the correctness of P’ with respect to T ‘’. ⬢ Create T ‘’’, a new test suite for P ‘, from T, T ‘p , and T ‘’. 28  05-09-2024 References ⬢ Software Testing: Principles and Practices. Naresh Chauhan, Oxford University Press, Second Edition, 2016 ⬢ Fundamentals of Software Engineering. Rajib Mall, PHI Learning Private Limited, 2018 29