Software Testing PDF

Software Testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 1 Outline  Terminology  System testing  Types of errors ...

Software Testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 1 Outline  Terminology  System testing  Types of errors  Function testing  Structure Testing  Dealing with errors  Performance testing  Quality assurance vs Testing  Acceptance testing  Component Testing  Installation testing  Unit testing  Integration testing  Testing Strategy  Design Patterns & Testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 2 Some Observations  It is impossible to completely test any nontrivial module or any system  Theoretical limitations: Halting problem ??  Practial limitations: Prohibitive in time and cost  Testing can only show the presence of bugs, not their absence (Dijkstra) total number of execution paths? loop 200 times Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 3  Testing Activities Subsystem Requirements Unit System Code Test Analysis Design Document Tested Document User Subsystem Subsystem Manual Unit Code Test Tested Integration Functional Subsystem Test Test Integrated Functioning Subsystems System Tested Subsystem Subsystem Unit Code Test All tests by developer Cf. levels of testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 4  Testing Activities continued Client’s Global Understanding User Requirements of Requirements Environment Validated Accepted Functioning System System PerformanceSystem Acceptance Installation Test Test Test Usable Tests by client System Tests by developer User’s understanding System in Use Tests (?) by user Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 5 Level of abstraction Levels of Testing in V Model system system requirements integration software acceptance requirements test preliminary software design integration detailed component design test code & unit debug test Time N.B.: component test vs. unit Bernd Bruegge & Allen H. Dutoit test; acceptance test vs. system integration Object-Oriented Software Engineering: Using UML, Patterns, and Java 6  Test Planning  A Test Plan:  A test plan includes:  covers all types and phases of  test objectives testing  schedule and logistics  guides the entire testing process  test strategies  who, why, when, what  test cases  developed as requirements,  procedure functional specification, and high-  data level design are developed  expected result  should be done before implementation starts  procedures for handling problems Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 7 Fault Handling Techniques Fault Handling Fault Tolerance Fault Avoidance Fault Detection Design Atomic Modular Reviews Methodology Transactions Redundancy Configuration Verification Management Testing Debugging Unit Integration System Correctness Performance Testing Testing Testing Debugging Debugging Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 8 Quality Assurance encompasses Testing Quality Assurance Usability Testing Scenario Prototype Product Testing Testing Testing Fault Avoidance Fault Tolerance Atomic Modular Configuration Transactions Redundancy Verification Management Fault Detection Reviews Debugging Walkthrough Inspection Testing Correctness Performance Unit Integration System Debugging Debugging Testing Testing Testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 9 Types of Testing  Unit Testing:  Individual subsystem  Carried out by developers  Goal: Confirm that subsystems is correctly coded and carries out the intended functionality  Integration Testing:  Groups of subsystems (collection of classes) and eventually the entire system  Carried out by developers  Goal: Test the interface among the subsystem Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 10 System Testing Terminology:  System Testing: system testing here = validation testing  The entire system  Carried out by developers  Goal: Determine if the system meets the requirements (functional and global)  Acceptance Testing: 2 kinds of Acceptance testing  Evaluates the system delivered by developers  Carried out by the client. May involve executing typical transactions on site on a trial basis  Goal: Demonstrate that the system meets customer requirements and is ready to use  Implementation (Coding) and testing go hand in hand Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 11 Unit Testing  Informal:  Incremental coding Write a little, test a little  Static Analysis:  Hand execution: Reading the source code  Walk-Through (informal presentation to others)  Code Inspection (formal presentation to others)  Automated Tools checking for  syntactic and semantic errors  departure from coding standards  Dynamic Analysis:  Black-box testing (Test the input/output behavior)  White-box testing (Test the internal logic of the subsystem or object)  Data-structure based testing (Data types determine test cases) Which is more effective, static or dynamic analysis? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 12 Black-box Testing  Focus: I/O behavior. If for any given input, we can predict the output, then the module passes the test.  Almost always impossible to generate all possible inputs ("test cases") why?  Goal: Reduce number of test cases by equivalence partitioning:  Divide input conditions into equivalence classes  Choose test cases for each equivalence class. (Example: If an object is supposed to accept a negative number, testing one negative number is enough)  If x = 3 then …  If x > -5 and x < 5 then … What would be the equivalence classes? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 13 Black-box Testing (Continued)  Selection of equivalence classes (No rules, only guidelines):  Input is valid across range of values. Select test cases from 3 equivalence classes:  Below the range Are these complete?  Within the range  Above the range  Input is valid if it is from a discrete set. Select test cases from 2 equivalence classes:  Valid discrete value  Invalid discrete value  Another solution to select only a limited amount of test cases:  Get knowledge about the inner workings of the unit being tested => white-box testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 14 White-box Testing  Focus: Thoroughness (Coverage). Every statement in the component is executed at least once.  Four types of white-box testing  Statement Testing  Loop Testing  Path Testing  Branch Testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 15 White-box Testing (Continued)  Statement Testing (Algebraic Testing): Test single statements  Loop Testing:  Cause execution of the loop to be skipped completely. (Exception: Repeat loops)  Loop to be executed exactly once  Loop to be executed more than once  Path testing:  Make sure all paths in the program are executed  Branch Testing (Conditional Testing): Make sure that each possible outcome from a condition is tested at least once if ( i = TRUE) printf("YES\n");else printf("NO\n"); Test cases: 1) i = TRUE; 2) i = FALSE Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 16 White-Box Testing Loop Testing [Pressman] Simple loop Nested Loops Concatenated Loops Unstructured Loops Why is loop testing important? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 17 Comparison of White & Black-box Testing  White-box Testing:  Both types of testing are needed  Potentially infinite number of  White-box testing and black box paths have to be tested testing are the extreme ends of a  White-box testing often tests testing continuum. what is done, instead of what should be done  Any choice of test case lies in between and depends on the  Cannot detect missing use cases following:  Black-box Testing:  Number of possible logical paths  Potential combinatorical  Nature of input data explosion of test cases (valid & invalid data)  Amount of computation  Often not clear whether the  Complexity of algorithms and selected test cases uncover a data structures particular error  Does not discover extraneous use cases ("features") Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 18  The 4 Testing Steps 1. Select what has to be 3. Develop test cases measured  A test case is a set of test  Analysis: Completeness of data or situations that will be requirements used to exercise the unit (code, module, system) being  Design: tested for cohesion tested or about the attribute  Implementation: Code tests being measured 2. Decide how the testing is 4. Create the test oracle done  An oracle contains of the  Code inspection predicted results for a set of  Proofs (Design by Contract) test cases  Black-box, white box,  The test oracle has to be written down before the  Select integration testing actual testing takes place strategy (big bang, bottom up, top down, sandwich) Next module Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 19 Guidance for Test Case Selection  Use analysis knowledge  Use implementation about functional knowledge about algorithms: requirements (black-box testing):  Examples:  Use cases  Force division by zero  Expected input data  Use sequence of test cases for interrupt handler  Invalid input data  Use design knowledge about system structure, algorithms, data structures (white-box testing):  Control structures  Test branches, loops,...  Data structures  Test records fields, arrays,... Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 20 Unit-testing Heuristics 1. Create unit tests as soon as object 4. Desk check your source code design is completed:  Reduces testing time  Black-box test: Test the use 5. Create a test harness cases & functional model  Test drivers and test stubs are  White-box test: Test the needed for integration testing dynamic model 6. Describe the test oracle  Data-structure test: Test the  Often the result of the first object model successfully executed test 2. Develop the test cases 7. Execute the test cases  Goal: Find the minimal  Don’t forget regression testing number of test cases to cover  Re-execute test cases every time as many paths as possible a change is made. Big cost -> what should be done? 3. Cross-check the test cases to eliminate duplicates 8. Compare the results of the test with the test oracle  Don't waste your time!  Automate as much as possible Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 21  Who Tests the Software? developer independent tester Understands the system Must learn about the system, but, will test "gently" but, will attempt to break it and, is driven by "delivery" and, is driven by quality Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 22 Summary  Testing is still a black art, but many rules and heuristics are available  Testing consists of component-testing (unit testing, integration testing) and system testing, and …  OOT and architectural testing, still challenging  User-oriented reliability modeling and evaluation not adequate  Testing has its own lifecycle Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 23  Terminology  Reliability: The measure of success with which the observed behavior of a system confirms to some specification of its behavior.  Failure: Any deviation of the observed behavior from the specified behavior.  Error: The system is in a state such that further processing by the system will lead to a failure.  Fault (Bug): The mechanical or algorithmic cause of an error. There are many different types of errors and different ways how we can deal with them. Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 24 Examples of Faults and Errors  Faults in the Interface  Mechanical Faults (very specification hard to find)  Mismatch between what the  Documentation does not client needs and what the match actual conditions or server offers operating procedures  Mismatch between  Errors requirements and implementation  Stress or overload errors  Capacity or boundary errors  Algorithmic Faults  Timing errors  Missing initialization  Throughput or performance  Branching errors (too soon, errors too late)  Missing test for nil Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 25  Dealing with Errors  Verification:  Assumes hypothetical environment that does not match real environment  Proof might be buggy (omits important constraints; simply wrong)  Modular redundancy:  Expensive  Declaring a bug to be a “feature”  Bad practice  Patching  Slows down performance  Testing (this lecture)  Testing is never good enough Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 26 Another View on How to Deal with Errors  Error prevention (before the system is released):  Use good programming methodology to reduce complexity  Use version control to prevent inconsistent system  Apply verification to prevent algorithmic bugs  Error detection (while system is running):  Testing: Create failures in a planned way  Debugging: Start with an unplanned failures  Monitoring: Deliver information about state. Find performance bugs  Error recovery (recover from failure once the system is released):  Data base systems (atomic transactions)  Modular redundancy  Recovery blocks Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 27 What is this? A failure? An error? A fault? Need to specify the desired behavior first! Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 28 Erroneous State (“Error”) Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 29 Algorithmic Fault Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 30 Mechanical Fault Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 31  How do we deal with Errors and Faults? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 32 Verification? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 33 Modular Redundancy? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 34 Declaring the Bug as a Feature? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 35 Patching? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 36 Testing? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 37 Testing takes creativity  Testing often viewed as dirty work.  To develop an effective test, one must have:  Detailed understanding of the system  Knowledge of the testing techniques  Skill to apply these techniques in an effective and efficient manner  Testing is done best by independent testers  We often develop a certain mental attitude that the program should in a certain way when in fact it does not.  Programmer often stick to the data set that makes the program work  "Don’t mess up my code!"  A program often does not work when tried by somebody else.  Don't let this be the end-user. Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 38 Test Cases  Test case 1 : ? (To execute loop exactly once)  Test case 2 : ? (To skip loop body)  Test case 3: ?,? (to execute loop more than once)  These 3 test cases cover all control flow paths Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 39