Unit Testing and Test-Driven Development (CSCI 2134) PDF

https://xkcd.com/1700/ 1  Unit Testing and Test-Driven Development CSCI 2134: Software Development Agenda Lecture Content Creating unit tests Basis testing Path testing Requirements testing Test Driven Development...

https://xkcd.com/1700/ 1  Unit Testing and Test-Driven Development CSCI 2134: Software Development Agenda Lecture Content Creating unit tests Basis testing Path testing Requirements testing Test Driven Development Brightspace Quiz Readings: This Lecture: Chapter 22 Next Lecture: Chapter 22 3 Creating Unit Tests A unit tests is targeted at small pieces of code written by a single programmer: Methods Classes Unit test are used to ensure that: Each method works: given the parameters, and a known state of the object on which the method is called, the method does what is expected The class works: a sequence of operations on an object of the class does what is expected 4 Examples of Functionality Tested by Unit Tests Methods Classes Example 1: Queue.size() Example 1: A Queue class The method returns the size of the A sequence of items is dequeued in queue the same order they were enqueued. Example 2: Matrix.clone() Number of items in the queue is equal to number of items enqueued The method returns a copy of the minus number of items dequeued matrix Example 2: A Matrix class Note: Sometimes it’s hard to Operations performed on a Matrix distinguish between method unit object are not lost tests and class unit tests (that’s ok) 5 Unit Test Deliverables What are you being asked to do? Create tests for each method that you write Create tests for each class that you write Questions: How many tests do we need to create? How long should the tests be? How exhaustive should the tests be? Should they be large tests or small tests? How do we know when we have enough tests? All these questions ask the same thing: What makes a good test suite? 6 What makes a good test suite? Good test suites: Good (high) coverage Tests all (most) parts of the code we are testing Types of Coverage Functions / methods: Every method gets called by a test Statements: Every statement gets executed by a test Branches / Loop (if / switch / for / while statements): Every branch gets taken Path coverage: Every path through the code is performed Smallest number of tests necessary Minimize the overlap Tests take time to write, run, and verify Smaller tests are easier to understand and debug Include all boundary conditions and exceptional cases Can be hard to write 7 Testing Pitfalls Create “clean” tests Wrong mindset: developers want to verify that code works instead of thinking how to break the code Optimistic view of coverage Common to assume 95% coverage, when the true figure is between 30% - 80% Oversimplification Assume the statement coverage is sufficient Do not do component or integration testing because such tests are harder to design and implement Need a structured approach to avoiding these pitfalls 8 A Structured Approach to Writing Unit Tests Task: Enumerate and identify all parts of our code we want to test White Box Approaches Control-flow Structured basis testing (exercise each statement) Path testing (exercise each path) Data-flow Black Box Approaches Requirements testing 9 How many tests do we need? Naive Code block M First test case code: code block M if (A or B or C) code block N Code block N code block P code block Q else code block S Code block O code block T Code block P code block U if (D or E) Second test case code: Code block Q code block M code block O else code block P Code block R code block R code block S Code block S code block U for (loop conditions) Code block T Two test cases exercise every statement of code Assume Code blocks have no branches or loops Code block U Doesn’t test extra conditions 10 Structured Basis Testing Idea: For each method create tests that test each part of the code in the method Each part of the code needs to be tested at least once Algorithm Start with one test case at start of method Add a test case for each Loop If statement Part of a Boolean condition Every code branch Observation: Smaller methods need fewer test cases 11 Warm up. Code block M Test Case 1 Test Case 5 Code Block M Code Block M if A Code Block N Code Block S Code block N Code Block S Test Case 2 else if B Code Block M Code block P Code Block P else if C Code Block S Test Case 3 Code block Q Code Block M else if D Code Block Q Code Block S Code block R Test Case 4 Code block S Code Block M Code Block R Code Block S 12 Structured Basis Testing: How many test? One: Four: Seven: code block M code block M code block M Code block M code block O code block C->N code block O if (A or B or C) code block P code block P code block P code block R code block R code block R Code block N code block S code block S code block S else code block U code block U code block T Code block O Two: Five: code block U code block M code block M Code block P code block A->N code block O if (D or E) code block P code block P code block R code block D->Q Code block Q code block S code block S else code block U code block U Three: Six: Code block R code block M code block M Code block S code block B->N code block O for (loop conditions) code block P code block P code block R code block E->Q Code block T code block S code block S Code block U code block U code block U 13 Compound Boolean Expressions OR Code block M Code block M 4 tests cases: 4 tests cases: if A if (A or B or C) 1. M, O, S 1. M, O, S 2. M, A->N, S Code block N1 2. M, N1, S Code block N 3. M, B->N, S 3. M, N2, S else if B else 4. M, C->N, S 4. M, N3, S Code block N2 Code block O else if C Code block S Code block N3 else Code block O Code block S 14  Compound Boolean Expressions AND Code block M 4 tests cases: Code block M 4 tests cases: 1. M, N, S 1. M, N, S if (A) if (A and B and C) 2. M, O1, S 2. M, (~C)->O, S if (B) 3. M, O2, S Code block N 3. M, (~B)->O, S if (C) 4. M, O3, S 4. M, (~A)->O, S else Code block N else Code block O Code block O1 Code block S else Code block O2 else Code block O3 Code block S 15 Example: How many tests do we need? public Matrix multiplyWithScalar(double s, Matrix res) { if (res != null) { if ((res.getHeight() 3 != height) || Note: two more tests cases (res.getWidth() 4 != width)) { are possible if height or return null; width can be zero. } 1 2 // assume height, width > 0 for (int i = 0; i < height; i++) { for (int j = 0; i < width; j++) { res.matrix[i][j] = s * matrix[i][j]; } } Test Cases: } 1. res == null; expected return: null 2. res.height == height, res.width = width, expected return s*this return res; 3. res != null, res.height != height, expected return null } 4. res.height == height, res.width != width, expected return null 16 Path Testing Idea: Instead of just ensuring that each How to avoid? Check all four cases statement is executed, we want to ensure temp < 0, time = 0, time 60 Same statement may do different things temp >= 0, time > 60 depending on the path! Key Idea: Example: In structured basis testing we sum the if (temp < 0) { number of test cases day = null; In path testing we multiply the number of } test cases if (time > 60) { day.getCoffee(); } Will this statement work? Depending on the structured basis test, some bugs can be missed! 17 How many tests do we need for path testing of just code blocks? Code block M x1 Paths if (A or B or C) M, N, P, Q, S, U Code block N M, N, P, Q, S, T, U else x2 M, N, P, R, S, U Code block O M, N, P, R, S, T, U Code block P x1 M, O, P, Q, S, U if (D or E) M, O, P, Q, S, T, U M, O, P, R, S, U Code block Q x2 M, O, P, R, S, T, U else Total # of paths = 1 x 2 x 1 x 2 x 2(+) x 1 = 8(+) Code block R Why 2+ for the loop? Code block S x1 Depends on whether the number of for (loop conditions) iterations matters x2+ Code block T Code block U x1 18 How many tests do we need for path testing? Code block M x1 Paths if (A or B or C) M, Nx3, P, Qx2, S, U Code block N M, Nx3, P, Qx2, S, T, U else x4 M, Nx3, P, R, S, U Code block O M, Nx3, P, R, S, T, U Code block P x1 M, O, P, Qx2, S, U if (D or E) M, O, P, Qx2, S, T, U M, O, P, R, S, U Code block Q x3 M, O, P, R, S, T, U else Total # of paths = 1 x 4 x 1 x 3 x 2(+) x 1 = Code block R 24(+) Code block S x1 Why 2+ for the loop? for (loop conditions) Depends on whether the number of x2+ Code block T iterations matters Code block U x1 19 How about this for path testing? Code block M x1 Test Case 1 Code Block M if A Notice, in this case, this is Code Block N Code block N Code Block S the same 5 test cases as Test Case 2 structured basis testing. else if B Code Block M Code block P Code Block P else if C x5 Code Block S Test Case 3 Code block Q Code Block M else if D Code Block Q Code Block S Code block R Test Case 4 Code block S x1 Code Block M Code Block R Code Block S Q: Why 5 and not 4? Test Case 5 A: There is an implicit else Code Block M Code Block S 20 Structured Basis vs Path Testing Structured Basis Testing: 3 cases Path Testing: 4 cases Code block M 1. M, P, R, S 1. M, P, R, S 2. M, N, R, S 2. M, N, R, S if A 3. M, P, Q, S 3. M, N, Q, S Code block N 4. M, P, Q, S else Code block P if B Code block Q else Code block R Code block S 21 Combinatorial Coverage Idea: We want to ensure that all Even more cases possible combinations of conditions are temp < 0, time = 0, time 60, day != null Same path may do different things temp >= 0, time > 60, day != null depending on the condition! Example: temp < 0, time = 0, time 60, day == null } temp >= 0, time > 60, day == null if (time > 60 || day != null) { day.getCoffee(); } For N conditions we have 2N combinations. 22 Motivation for Requirements/Specification Testing Tends to be preferred… why? Can be used with Test Driven Development Avoids implementation-dependent biases Supports standard design principles (SOLID) Forces the developer to understand the requirements Same tests can be reused for different implementations 23 Blackbox: Requirements/Specification Testing Idea: Use the specification to determine Error / Exceptional cases Boundary cases Common cases 24 Creating Test Cases Using Requirements or Specification Process: Create 1+ test cases for Each error condition E.g., returns null or throws exception Typically based on parameters and return values Ignore undefined behavior Each boundary condition E.g., minimum, maximum, and threshold values Typically based on method description Common / typical cases Can be very subjective Anything that is not an error or boundary 25  Postage data from Canada Post at https://www.canadapost.ca/cpo/mc/personal/ratesprices/postalprices.js on Sept. 2015 Weight Canada US Up to 30g $0.85 $1.20 Example: Postage Problem Over 30g and up to 50g $1.20 $1.80 Up to 100g $1.80 $2.95 /* @description: This method takes a Over 100g and up to 200g $2.95 $5.15 * destination country (Canada or US) Over 200g and up to 300g $4.10 $10.30 * and the weight of a standard envelope Over 300g and up to 400g $4.70 $10.30 * and returns the needed postage. Over 400g and up to 500g $5.05 $10.30 * If the weight is not positive or the * country is invalid, -1 is returned. Error cases: * weight test.$T.out Output file if diff test.$T.out test.$T.gold ; then echo " " PASSED else Expected output file echo " " FAILED fi Compare outputs Input file done 34 Test Harnesses and Stubs Test harnesses and stubs provide a way to test a component of a piece of software Test Environment Test harness Test Harness Makes calls to the code being tested Typically has a main-line program May or may not be driven by external input E.g., a Runner program that you may have implemented in CSCI 1110 Stub Small pieces of code that simulate code that your component may call while it is running Typically used when the actual code is too complex, unfinished, or unpredictable to be provide known return values Stub Stub Stub E.g., Using empty methods when implementing a class 1 2 3 In many cases a combination of test harness, stubs, and scripts are used 35 Example of a Test Harness 36 Testing Frameworks Test Environment The test-harness approach is so common that special Test Code libraries exist to provide test harnesses test1() test2() The programmer provides methods, each of which … perform a test on the target code testX() This requires the programmer to focus on the tests instead of writing code to support the tests Examples of Testing Frameworks Artos Arquillian AssertJ beanSpec BeanTest Cactus Concordion Concutest Cucumber-JVM Cuppa DbUnit EasyMock EtlUnit EvoSuite GrandTestAuto GroboUtils HavaRunner Instinct Java Server-Side Testing framework (JSST) JBehave JDave JExample JGiven JMock JMockit Jnario Jtest Jukito JUnit JUnitEE JWalk Mockito Mockrunner Needle NUTester OpenPojo PowerMock Randoop Spock SpryTest SureAssert Tacinga TestNG Unitils XMLUnit 37 https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#Java  Example of JUnit Test for a Matrix class import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; import java.util.Scanner; Regular Java class Input for the tests Test getElem() class MatrixTest { private final static String simpleMatrix = "2 2 1 2 3 4"; // [ 1 2 ] // [ 3 4 ] @Test Each method is a void getElem() { test Matrix m = new Matrix(new Scanner(simpleMatrix)); assertEquals(2, m.getElem(1,2), "getElem() did not return correct value"); } @Test Test the getElem() method Create a Matrix object void setElem() { Test setElem() Matrix m = new Matrix(new Scanner(simpleMatrix)); m.setElem(2, 1, 5); assertEquals(5, m.getElem(2,1), "setElem() may not have set correct value"); } 38 Key Points Testing is a set of techniques that should be used in combination to detect defects Testing is challenging, requiring a different mindset but cannot verify that a piece of software is defect free Testing is performed at various granularities such as unit, component, integration, and system Blackbox testing is strictly based on the specification while whitebox testing incorporates knowledge of the implementation Regression testing reruns past tests to confirm the code was not broken It is beneficial to create and test in the course of development to catch defects as quickly as possible 39 Image References Retrieved January 8, 2020 http://pengetouristboard.co.uk/vote-best-takeaway-se20/ https://i.pinimg.com/originals/b5/22/38/b52238fad11b0a3ecac36fa17604 1d98.jpg https://www.nbs-system.com/wp- content/uploads/2016/05/160503_Tests_boites-788x433.jpg https://www.pinclipart.com/picdir/middle/29-293393_challenges-peace- first-clip-art-work-teamwork-clip.png https://miro.medium.com/max/10000/1*6HUPtGBOSdERQkCOpL9QEg.pn g https://pngtree.com/freepng/black-repeat-icon_4841007.html Retrieved September 2, 2020 https://dzone.com/articles/software-testing-comic 40 LOOK, THE LATENCY FALLS EVERY TIME YOU CLAP YOUR HANDS AND SAY YOU BELIEVE 1  Debugging CSCI 2134: Software Development Agenda Lecture Contents Motivation Debugging with the scientific method Using a debugger Fixing the bug Brightspace Quiz Readings: This Lecture: Chapter 23 Next Lecture: Chapter N/A (Common Coding Mistakes) 3 Why Debug? What is debugging? Debugging is the process of locating and correcting the root cause of defects Why debug? Because we do not write perfect code How do we know we have a bug? Our program exhibits symptoms indicating there is a problem What do we have to do? 1. Find the root cause of the bug (a.k.a defect) 2. Fix the root cause What should we not do? Fix the symptom instead of the root cause 4 Bugs are Opportunities to Learn about … (McConnel, CC2, 2004) The program you’re working on The kinds of mistakes you make The quality of your code from the point of view of someone who has to read How you solve problems How you fix defects 5 How Not to Debug (McConnel, CC2, 2004) Find the defect by guessing Original code: Don’t spend time to understand int sum = 0; the problem for (int i = 0; i