03_testing.pdf

SOEN 6441 Lecture 3 Verification & Validation Testing Fundamentals Introduction...

SOEN 6441 Lecture 3 Verification & Validation Testing Fundamentals Introduction Definitions The V Model Black Box vs. White Box SOEN 6441 Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading SOEN 6441 Department of Computer Science and Software Engineering Concordia University 3.1 Outline SOEN 6441 Verification & Validation Introduction Definitions 1 Verification & Validation The V Model Black Box vs. White Box Introduction Unit Testing Equivalence Classes Definitions Testing Strategies The V Model Notes and Further Reading Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies 2 Notes and Further Reading 3.2  SOEN 6441 Two key SQA concerns Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading Software verification: Software validation: - “Are we building the product right?” - “Are we building the right product?” 3.3  SOEN 6441 Verification and Validation (V&V) techniques come in two varieties Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading Static V&V: Dynamic V&V: - Analyses of the code - Executing the code 3.4  SOEN 6441 What is testing? Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Testing is a formal process carried out by a Equivalence Classes Testing Strategies specialized testing team in which a software unit, Notes and Further Reading several integrated software units, or an entire software system are examined by running the programs on a computer. All the associated tests are performed according to approved test procedures on approved test cases. 3.5  SOEN 6441 Basic testing definitions Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Errors, faults, failures, and incidents: Unit Testing Equivalence Classes Testing Strategies - Errors are mistakes that are made by people Notes and Further Reading - Faults are the result of an error in software artifacts - Failures occur when a fault executes - Incidents are consequences of failures - Software testing exercises the software with test cases to find faults or gain confidence in the system (execution based on test cases) 3.6  SOEN 6441 Basic testing definitions Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Test cases, stubs, and drivers: Unit Testing Equivalence Classes Testing Strategies - Test cases provide a set of inputs and list a set of Notes and Further Reading expected outputs - Test stubs are partial implementations (simulation) of components on which a component under test depends - Test drivers are partial implementations of components that exercise and depend on the tested component 3.7  SOEN 6441 An overview of testing definitions Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Test suite Equivalence Classes Testing Strategies exercises is revised by Notes and Further Reading * * 1…n * * Test case Component Correction * * * Test stub finds repairs * Test driver * * Failure * * Fault * * Error is caused by is caused by 3.8  SOEN 6441 What does he mean? Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies E. Dijkstra “Program testing can be Notes and Further Reading a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.” 3.9  SOEN 6441 The ugly truth about software testing Verification & Validation Introduction Definitions Bugs are inescapable: The V Model Black Box vs. White Box Unit Testing - It is impractical to test all operating conditions Equivalence Classes Testing Strategies - Testing, although incomplete, increases our Notes and Further Reading confidence that software has its desired behaviour The impracticality of complete testing: - Large input space - Large output space - Large state space - Large number of possible execution paths - Subjective requirements 3.10  SOEN 6441 The ugly truth about testing: Large input and state spaces Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Exercise: How many tests do we need to Unit Testing Equivalence Classes Testing Strategies exhaustively test: int testMe(int x, int y) {... } Notes and Further Reading Exhaustive testing: - Testing all possible input combinations is often infeasible 3.11  SOEN 6441 The ugly truth about testing: Large input and state spaces Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Exhaustive testing: Unit Testing Equivalence Classes Testing Strategies - It gets worse! Notes and Further Reading - Imagine that you need to test the Java compiler - Exhaustive testing implies that you would need to try all possible Java programs! - At best, exhaustive testing is impractical, at worst it’s impossible! 3.12  SOEN 6441 Exercise: How many execution paths? Verification & Validation Introduction Definitions... The V Model Black Box vs. White Box for (int i = 0; i < n; ++i) { Unit Testing if (a.get(i) == b.get(i)) Equivalence Classes Testing Strategies x[i] = x[i] + 100; Notes and Further else Reading x[i] = x[i]/2; }... 3.13  SOEN 6441 Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading 3.14  SOEN 6441 The ugly truth about testing: Large number of possible execution paths Verification & Validation Introduction Definitions... The V Model Black Box vs. White Box for (int i = 0; i < n; ++i) { Unit Testing if (a.get(i) == b.get(i)) n Number of Equivalence Classes Testing Strategies If each test takes 10-31 seconds, x[i] = x[i] + 100; else 2 withpaths Notes and Further Reading x[i] = x[i]/2; n=36, we need more2 time than }... 4 has passed since the3 big bang! Loop Header 8 10 1024 20 1048576 + 100 /2 60 1.15 x 1018 Number of Paths = 2n 3.15  SOEN 6441 Quality assurance is a risk mitigation exercise Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading Striking a balance: - A balance must be reached between the cost of testing and the risk of missing bugs 3.16  SOEN 6441 The ugly truth about testing: Bugs, bugs, bugs Verification & Validation Introduction Definitions The V Model The truth about bugs: Black Box vs. White Box Unit Testing Equivalence Classes - Not all known bugs are fixed Testing Strategies Notes and Further - Not enough time Reading - Too risky to fix (might create more problems) - When is a bug really a bug? - Time pressure may lead to downgrading bugs to feature requests - Since many people struggle to process criticism, QA personnel are not the most popular people  3.17  SOEN 6441 V The V model of development Verification & Validation Introduction Definitions The V Model Acceptance requirements acceptance testing Black Box vs. White Box Test Plan Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading System architecture Test Plan system testing Integration component design Test Plan integration testing Unit module design Test Plan unit testing implementation 3.18  SOEN 6441 Differences between the testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing system Equivalence Classes unit testing integration Testing Strategies testing testing Notes and Further Reading from module from interface from requirements specifications specifications specifications visibility visibility of no visibility of of code details integr. structure code complex some no drivers/stubs scaffolding scaffolding behavior of single interactions system modules among modules functionalities 3.19  SOEN 6441 Testing visibility specifiers Verification & Validation Introduction Definitions The V Model Black-box testing: Black Box vs. White Box Unit Testing Equivalence Classes - Tests that are written without considering how Testing Strategies Notes and Further the code is written/structured (based on specs) Reading - Treats the system like an opaque box that accepts inputs and checks outputs White-box testing: - Tests that are written with an understanding of the structure of the code - Tests the inner workings of the system itself 3.20  SOEN 6441 Black-box or white-box? Verification & Validation Introduction Definitions The V Model Black Box vs. White Box (1) Unit tests? Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading (2) System tests? (3) Stress tests? (4) Integration tests? (5) Acceptance tests? 3.21  SOEN 6441 Black-box or white-box? Verification & Validation Introduction Definitions The V Model Black Box vs. White Box (1) Unit tests? White box Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading (2) System tests? Black-box (3) Stress tests? Black-box (4) Integration tests? White-box (5) Acceptance tests? Black-box 3.21  SOEN 6441 Black-box vs. White-box Verification & Validation Introduction Definitions The V Model Black-box: White-box: Black Box vs. White Box Unit Testing + Check conformance + Allows for measuring Equivalence Classes Testing Strategies Notes and Further with spec “test coverage” Reading + Scales up + Based on control or - Depends on detail data flow coverage in spec - Doesn’t scale up well - Does not tell how - Cannot reveal missing much of the system functionality! is being tested (coverage) 3.22  SOEN 6441 Do we need both types of tests? Verification & Validation Introduction Definitions The V Model system Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies specification Notes and Further Reading implementation Missing functionality cannot Unexpected functionality be revealed by white-box cannot be revealed by black-box techniques techniques 3.23  SOEN 6441 Which input set will find the bug? Verification & Validation Introduction Definitions The V Model Black Box vs. White Box... Unit Testing if x > y then Equivalence Classes Testing Strategies Max := x; Notes and Further else Reading Max := x; end if;... Input sets: {x=3, y=2; x=2, y=3} {x=3, y=2; x=4, y=3; x=5, y=1} 3.24  SOEN 6441 Flow graph for white-box testing Verification & Validation Introduction Definitions The V Model To help the programmer to systematically test the Black Box vs. White Box Unit Testing code Equivalence Classes Testing Strategies Notes and Further Each branch in the code (such as if and while Reading statements) creates a node in the graph Read P Read Q IF P+Q > 100 THEN Print “Large” ENDIF If P > 50 THEN Print “P Large” ENDIF 25 3.25  SOEN 6441 Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Read P Unit Testing Equivalence Classes Read Q Testing Strategies IF P+Q > 100 THEN Notes and Further Reading Print “Large” ENDIF If P > 50 THEN Print “P Large” ENDIF 26 3.26  SOEN 6441 Flow graph for white-box testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies The testing strategy has to reach a targeted Notes and Further Reading coverage of statements and branches; the objective can be to: cover all possible statements cover all possible branches cover all possible paths 27 3.27  Cover all possible SOEN 6441 statements Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Read P Unit Testing Equivalence Classes Read Q Testing Strategies IF P+Q > 100 THEN Notes and Further Reading Print “Large” ENDIF If P > 50 THEN Print “P Large” ENDIF 1A-2C-3D-E-4G-5H 28 3.28  Cover all possible SOEN 6441 branches Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Read P Unit Testing Equivalence Classes Read Q Testing Strategies IF P+Q > 100 THEN Notes and Further Reading Print “Large” ENDIF If P > 50 THEN Print “P Large” ENDIF 1A-2C-3D-E-4G-5H 1A-2B-E-4F 29 3.29  Cover all possible SOEN 6441 paths Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Read P Unit Testing Equivalence Classes Read Q Testing Strategies IF P+Q > 100 THEN Notes and Further Reading Print “Large” ENDIF If P > 50 THEN Print “P Large” ENDIF 1A-2B-E-4F 1A-2B-E-4G-5H 1A-2C-3D-E-4G-5H 1A-2C-3D-E-4F 30 3.30  SOEN 6441 Testing stages Verification & Validation Introduction Definitions The V Model Development testing: Black Box vs. White Box Unit Testing - Tests that are typically written by system Equivalence Classes Testing Strategies Notes and Further designers and programmers Reading Release testing: - Separate QA team produces tests to check the system before it is released User testing: - Tests run to ensure that users will be satisfied with the product 3.31  SOEN 6441 Development testing Granularities Verification & Validation Introduction Definitions The V Model Unit testing: Black Box vs. White Box Unit Testing - Target: particular program unit (class) Equivalence Classes Testing Strategies Notes and Further - Focuses on testing the functionality of an Reading object or a method 3.32  SOEN 6441 Development testing Unit testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit testing: Unit Testing Equivalence Classes Testing Strategies - The name of the game is to maximize coverage Notes and Further Reading of the system while minimizing execution time - We want our test cases to cover all of the possible paths through our methods - However, we want our tests to finish quickly so that we don’t delay the release process! 3.33  SOEN 6441 Development testing Unit testing Verification & Validation Introduction Definitions The V Model Key phases: Black Box vs. White Box Unit Testing Equivalence Classes - Setup part: Prepares the system for executing Testing Strategies Notes and Further the test case Reading - Call part: Call the method under test - Assertion part: Check that the system state is in its expected form - Teardown part: Reset the system to its pre- setup state 3.34  SOEN 6441 Development testing Unit testing goals Verification & Validation Introduction Definitions The V Model Goals in designing unit tests: Black Box vs. White Box Unit Testing Equivalence Classes (1) Show that (when used as expected) the unit Testing Strategies Notes and Further does what it is supposed to do Reading (2) If there are defects in a unit, the test cases should reveal them! Need two types of unit tests to do this! 3.35  SOEN 6441 Development testing Unit testing strategies Verification & Validation Introduction Definitions The V Model Partition testing: Black Box vs. White Box Unit Testing Equivalence Classes - Identify groups of inputs that have common Testing Strategies Notes and Further characteristics (should be processed the same way) Reading Guideline-based testing: - Use guidelines based on previous experience of the kinds of errors that programmers often make - This means designing tests that explicitly try to catch a range of specific types of defects that commonly occur 3.36  SOEN 6441 Development testing Partition testing Verification & Validation Introduction Definitions The V Model Equivalence partitioning: Black Box vs. White Box Unit Testing Equivalence Classes - Inputs and outputs can be thought of as members Testing Strategies Notes and Further of sets with common characteristics Reading - Rule of thumb - Select test cases on the edge of the partitions - Select a test case near the middle as well 3.37  SOEN 6441 Equivalence classes Verification & Validation Introduction Definitions The V Model Black Box vs. White Box It is inappropriate to test by brute force, using every possible input Unit Testing Equivalence Classes value Testing Strategies Notes and Further Takes a huge amount of time Reading Is impractical Is pointless! You should divide the possible inputs into groups which you believe will be treated similarly by all algorithms. Such groups are called equivalence classes. A tester needs only to run one test per equivalence class The tester has to understand the required input, appreciate how the software may have been designed 38 3.38  SOEN 6441 Development testing Partition testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading 3.39  SOEN 6441 Examples of equivalence classes Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Valid input is a month number (1-12) Notes and Further Reading Equivalence classes are: [-∞..0], [1..12], [13.. ∞] Valid input is one of ten strings representing a type of fuel Equivalence classes are 10 classes, one for each string A class representing all other strings 40 3.40  SOEN 6441 Development testing Identifying partitions Verification & Validation Introduction Definitions “the program accepts four to ten inputs which The V Model Black Box vs. White Box are five-digit integers greater than 10,000.” Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading 3.41  SOEN 6441 Combinations of equivalence classes Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Combinatorial explosion means that you cannot Unit Testing Equivalence Classes realistically test every possible system-wide equivalence Testing Strategies Notes and Further class. Reading If there are 4 inputs with 5 possible values there are 5 4 (i.e. 625) possible system-wide equivalence classes. You should first make sure that at least one test is run with every equivalence class of every individual input. You should also test all combinations where an input is likely to affect the interpretation of another. You should test a few other random combinations of equivalence classes. 42 3.42 Example equivalence class SOEN 6441 combinations Verification & Validation Introduction Definitions The V Model One valid input is either ‘Metric’ or ‘US/Imperial’ Black Box vs. White Box Unit Testing Equivalence classes are: Equivalence Classes Testing Strategies Metric, US/Imperial, Other Notes and Further Reading Another valid input is maximum speed: 1 to 750 km/h or 1 to 500 mph Validity depends on whether metric or US/imperial Equivalence classes are: [-∞..0], [1..500], [501..750], [751.. ∞] Some test combinations Metric, [1..500] valid US/Imperial, [1..500] valid Metric, [501..750] valid US/Imperial, [501..750] invalid 43 3.43  SOEN 6441 Testing at boundaries of equivalence classes Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing More errors in software occur at the boundaries of Equivalence Classes Testing Strategies equivalence classes Notes and Further Reading The idea of equivalence class testing should be expanded to specifically test values at the extremes of each equivalence class E.g. The number 0 often causes problems E.g.: If the valid input is a month number (1-12) Test equivalence classes as before Test 0, 1, 12 and 13 as well as very large positive and negative values 44 3.44  SOEN 6441 Development testing Guideline-based testing Verification & Validation Introduction Definitions The V Model Guidelines for sequences, arrays, or lists: Black Box vs. White Box Unit Testing Equivalence Classes (1) Test software with sequences that have only one Testing Strategies Notes and Further value Reading - Common mistake! Assume sequences have multiple values! (2) Use different sequences of different sizes in different tests (increases coverage!) (3) Derive tests so that the first, middle, and last elements of the sequence are accessed (reveals partition boundaries!) 3.45  SOEN 6441 Development testing Guideline-based testing Verification & Validation Introduction Definitions The V Model More guidelines: Black Box vs. White Box Unit Testing Equivalence Classes - Choose inputs that force the system to generate all Testing Strategies Notes and Further error messages Reading - Design inputs that cause input buffers to overflow - Repeat the same input or series of inputs numerous times - Force invalid outputs to be generated - Force computation results to be too large or too small 3.46  SOEN 6441 Development testing Granularities Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Component (integration) testing: Unit Testing Equivalence Classes - Target: the combination of units that make up a Testing Strategies Notes and Further Reading component - Focuses on the interface that a component exposes to other components 3.47  SOEN 6441 Development testing Integration testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading 3.48  SOEN 6441 Development testing Types of component interfaces Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Parameter: Unit Testing Equivalence Classes - Passing parameters through method calls Testing Strategies Notes and Further Reading Shared memory: - Accessing a shared program element Procedural: - Interfaces in the typical (OO) sense Message passing: - A message is sent to request some service (e.g., client-server applications) 3.49  SOEN 6441 Development testing Types of interface errors Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Misuse: Unit Testing Equivalence Classes - Caller makes an error in the use of an interface Testing Strategies Notes and Further Reading Misunderstanding: - Caller makes an invalid assumption about the use of an interface Timing: - Some interfaces are time-sensitive - Even if a caller uses an interface correctly, it may be too early/late 3.50  SOEN 6441 Development testing Guidelines for component testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box (1) Examine the code to be tested and identify Unit Testing Equivalence Classes each call toreviews an external component Testing Strategies Code are great for Notes and Further Reading (2) Where objects are accepted as parameters, finding component/interface always test a null input (3) Design tests toproblems deliberately cause the component to fail (4) Where components use a shared memory interface, design tests to vary their order of access 3.51  SOEN 6441 Development testing Granularities Verification & Validation Introduction Definitions The V Model System testing: Black Box vs. White Box Unit Testing - Target: the combination of components that Equivalence Classes Testing Strategies Notes and Further make up a subsystem or the entire system Reading - Focuses on testing the interactions between components 3.52  SOEN 6441 Development testing System testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes How is it different from component testing? Testing Strategies Notes and Further Reading - All components are in the testing loop - Crosses team boundaries 3.53  SOEN 6441 Development testing Writing good system tests Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Use cases: Equivalence Classes Testing Strategies - Essentially, use cases describe expected system Notes and Further Reading behaviour - They make for an ideal place to start with your system tests Sequence diagrams can help to establish the expected system test behaviour! 3.54  SOEN 6441 Development testing: How much system testing is enough? Verification & Validation Introduction Definitions Policies: The V Model Black Box vs. White Box Unit Testing - Each project should outline testing policies to Equivalence Classes Testing Strategies reach the level of risk that is comfortable Notes and Further Reading Some example guidelines: (1) All system functions that are accessed through menus should be tested (2) Combinations of functions that are accessed through the same menu must be tested (3) Where user input is provided, all functions must be tested with correct and invalid input 3.55  SOEN 6441 Release testing Verification & Validation Introduction Definitions What is release testing? The V Model Black Box vs. White Box Unit Testing - Similar to system testing, release testing is Equivalence Classes Testing Strategies performed on the entire system and is based Notes and Further Reading on use cases - Unlike system tests, release tests: - Must be written by (and executed by) a team that does not report to development - Check that the system meets requirements (system tests are out to find bugs) 3.56  SOEN 6441 Release testing: How does it differ from system testing? Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Black-box vs. white box: Testing Strategies Notes and Further - Technically speaking, some development Reading organizations write white-box system tests - Release tests are never white-box tests, they are always black-box! 3.57  SOEN 6441 User testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes What are user tests? Testing Strategies Notes and Further - User tests are performed by customers to Reading ensure that the software meets their expectations - Useful approach for validation 3.58  SOEN 6441 User testing Types of user tests Verification & Validation Introduction Definitions The V Model Black Box vs. White Box 1. Alpha tests: Unit Testing Equivalence Classes - A select group of users works closely with the Testing Strategies Notes and Further development team to test early releases Reading - Defects are expected, but alpha users get access to “bleeding edge” features! 2. Beta tests: - Beta testing involves delivering a system to a number of potential customers who agree to use that system. - A larger group of users is allowed to experiment with a release 3.59  SOEN 6441 User testing Types of user tests Verification & Validation Introduction Definitions The V Model 3. Acceptance tests: Black Box vs. White Box Unit Testing Equivalence Classes - Customers test a software system to make sure Testing Strategies Notes and Further it meets their expectations and can be adopted Reading in their environment 3.60  SOEN 6441 Strategies for Testing Large Systems Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Big bang testing versus integration testing Equivalence Classes Testing Strategies Notes and Further In big bang testing, you take the entire system and test it Reading as a unit A better strategy in most cases is incremental testing: You test each individual subsystem in isolation Continue testing as you add more and more subsystems to the final product Incremental testing can be performed horizontally or vertically, depending on the architecture Horizontal testing can be used when the system is divided into separate sub-applications 61 3.61  SOEN 6441 Top down testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Start by testing just the user interface. Notes and Further Reading The underlying functionality are simulated by stubs. Pieces of code that have the same interface as the lower level functionality. Do not perform any real computations or manipulate any real data. Then you work downwards, integrating lower and lower layers. The big drawback to top down testing is the cost of writing the stubs. 62 3.62  SOEN 6441 Bottom-up testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Notes and Further Start by testing the very lowest levels of the software. Reading You needs drivers to test the lower layers of software. Drivers are simple programs designed specifically for testing that make calls to the lower layers. Drivers in bottom-up testing have a similar role to stubs in top-down testing, and are time-consuming to write. 63 3.63  SOEN 6441 Sandwich testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Sandwich testing is a hybrid between bottom-up and top Testing Strategies Notes and Further down testing. Reading Test the user interface in isolation, using stubs. Test the very lowest level functions, using drivers. When the complete system is integrated, only the middle layer remains on which to perform the final set of tests. 64 3.64  SOEN 6441 Vertical strategies for incremental integration testing Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Notes and Further Reading 65 3.65  SOEN 6441 The test-fix-test cycle Verification & Validation Introduction Definitions The V Model When a failure occurs during testing: Black Box vs. White Box Unit Testing Each failure report is entered into a failure tracking system. Equivalence Classes Testing Strategies It is then screened and assigned a priority. Notes and Further Reading Low-priority failures might be put on a known bugs list that is included with the software’s release notes. Some failure reports might be merged if they appear to result from the same defects. Somebody is assigned to investigate a failure. That person tracks down the defect and fixes it. Finally a new version of the system is created, ready to be tested again. 66 3.66  SOEN 6441 The ripple effect Verification & Validation Introduction Definitions The V Model There is a high probability that the efforts to remove Black Box vs. White Box Unit Testing the defects may have actually added new defects Equivalence Classes Testing Strategies Notes and Further The maintainer tries to fix problems without fully Reading understanding the ramifications of the changes The maintainer makes ordinary human errors The system regresses into a more and more failure- prone state 67 3.67  SOEN 6441 Regression testing Verification & Validation Introduction Definitions The V Model It tends to be far too expensive to re-run every single test Black Box vs. White Box Unit Testing case every time a change is made to software. Equivalence Classes Testing Strategies Hence only a subset of the previously-successful test cases is Notes and Further actually re-run. Reading This process is called regression testing. The tests that are re-run are called regression tests. Regression test cases are carefully selected to cover as much of the system as possible. The “law of conservation of bugs”: The number of bugs remaining in a large system is proportional to the number of bugs already fixed 68 3.68 Outline SOEN 6441 Verification & Validation Introduction Definitions 1 Verification & Validation The V Model Black Box vs. White Box Introduction Unit Testing Equivalence Classes Definitions Testing Strategies The V Model Notes and Further Reading Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies 2 Notes and Further Reading 3.69 Reading Material SOEN 6441 Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies Supplemental Notes and Further Reading [Som16, Chapter 8] (Software testing) 3.70 References SOEN 6441 Verification & Validation Introduction Definitions The V Model Black Box vs. White Box Unit Testing Equivalence Classes Testing Strategies [Som16] Ian Sommerville. Notes and Further Reading Software Engineering. Pearson, 10th edition, 2016. http://software-engineering-book.com. 3.71