Software Testing CS265 Lecture Notes PDF

SOFTWARE TESTING (CS265) W1-L2: Introduction to Software Testing Dr. Edgar Galván 1. Attendance in labs – via Moodle will record EOLAS IP Address to grant you marks 2. CA 15% + Exam 85% 3. I cannot access Moodle CS265 2 Upon completion of today’s lecture, you will be able to: know why Software Testing (ST) is so important know the differences between white-box testing and black-box testing know that ST is not perfect 3 1969: IBM announced separation of SW/HW1 1981: Software for the masses (PC) 1990s: Web 2000s: Mobile devices 2017: (close to) trillion $ industry 2 1 https://www-03.ibm.com/ibm/history/history/year_1969.html 2 http://www.softwaremag.com/a-digital-transformation/ 4 Still immature Software ß à Construction Software ß à Transportation Software ß à Medicine ... NIST (2002): ~ 60B $ loss per year (US)3 3Economic Impacts of Inadequate Infrastructure for ST, National Institute of Standards and Technology Report, 2002 5 Many models / techniques are applied Important for Quality Assurance Important for Software Engineering 'Software Crisis' (1960s/1970s) E.g. projects running over-budget, over-time, have low quality,... Software industry had to change Define roles of personnel Plan, document, manage the process See BOOK Section 1.1; P2 (TDD) 6 Software De-Bugging ß à ST ST from the 1970s From: Something to do after finishing the SW To: Something to do while creating SW Industry looked at benefits of ST as part of process Costs Performance Quality ... 7 Code is complex Requirements can be vague Pressure! Reduced quality à increased failure rate But also: Bad reputation, legal claims 8 ST is a process or strategy of finding, assessing, predicting defects in software products with the aim to reliably build large- scale and high quality applications. (see BOOK S1.1.2; P 4 – Cost of testing vs. profit) 9 § Instructions in separate programs that, when executed, provide a required functionality § Data structures § Configuration files § System documentation § Describes the structure of system § User documentation § Describes how to use the system § Design documentation § For software developers 10 Software testing includes the search for flaws in software (or their absence). Includes flaws in design documentation Three Types of flaws / defects: 1. Mistakes à software developer 2. Faults à in the code 3. Failures à incorrect (out-of-spec) behaviour See BOOK S1.2; P4-5 11 Algorithmic: Input-Output mapping with program is wrong Computation and Precision: Calculated result does not with expected accuracy. Syntax: Violates programming language Documentation that is incomplete/incorrect Capacity/boundary: System fails when fully loaded / heavily used (à large scale/quality argument) … See BOOK S1.2.2; P. 5 12 HP Study: 50% are algorithmic or computation / precision 13 See: Pfleeger: Software Engineering: Theory & Practice 4th Edition (2010) Severity Level Behaviour 1 (most severe) System crash with expensive recovery time; loss of function/data; no workaround 2 Partial loss of function or data; small workaround possible 3 Loss of function or data with manual workaround 4 (least severe) Cosmetic / minor inconvenience See BOOK S1.2.3; P6 14 Development (of the entire project up to that point): $7 billion This rocket + cargo: ~$500 million. link. 15 What happened? A software update was applied on 19th of June, that controlling payment processing. Wages, payments and transactions (incl. ATM cash withdrawal) were disrupted. Only resolved on 16th of July. Cause: Failure in a piece of batch software Cost: Unknown (Fine: £56M) Video 16 Business Computing World, 2012 (Video) Failure Curve for Hardware WHY? ST reduces risks in software development ST allows software to scale while maintaining quality ST reduces the risk of product failures Signs of SW failure: Not ready in time Costs over-run (software is becoming too costly to complete) SW does not solve users' problem SW has poor quality SW is hard to maintain (Example?) 21 Why does software have faults? It is hard... ... to collect requirements ... to specify SW behaviour ... to design SW ... to implement SW ... to modify SW... correctly. There are two engineering approaches: 1. Forward engineering 2. Feedback-based 22 23 In practice, all steps must be checked to ensure their correctness and mistakes are fixed Each step needs checking against specifications 24 We need to check if the software product meets users’ needs 25 Specifications are key for testing Represents a 'model' They define SW behaviour Specifications describe Normal behaviour of SW Error behaviour of SW 'Reasonable' behaviour has issues What is a reasonable behaviour if a parameter is missing from a function call? Ignore or Exception? What is a reasonable return value for a method that calculates temperature? Celsius or Fahrenheit? 26 So, specifications are necessary for testing. The tester needs to... Read and understand (user) requirements (expressed in English) and... ... turn them into a structured form (e.g. a table). This course uses a lot of tables. 27 A program 'check' for an online shop determines whether a customer gets a discount, based on points to date and whether the customer is a gold member. Bonus points accumulate as customers make purchases. Gold customers will get a discount when they accumulate 80 points. Other customers only get a discount once they have more than 120 points. All customers always have at least one point. 28 FULLPRICE: customer needs to pay in full DISCOUNT: customers gets discount ERROR: input parameter invalid (bonusPoints < 1) 29 See BOOK S1.4; P10 30 Why do we need a theory of ST? So, we can find a minimum set of test data that ensures that the software works for all inputs 1. In a successful test all test data in a test set must produce results as defined in specification 2. Test data selection criteria is reliable if it consistently produces test data that is (not) successful 3. If test data chosen with a reliable and valid criterion, then successful test means that program produces correct results for all input domain 4. Result: Only criterion that guarantees to be reliable and valid is the one that selects each and every value in the program domain à exhaustive testing 31 Test execution time It just takes a long time to run every possible test Test design time We still need to make sure that every result is correct 32 points goldCustomer Datatype? Long Datatype? Boolean Value range: 264 Value range: {true, false} = 2 265 36,893,488,147,419,103,232 33 NO WAY! We need test heuristics that simplify this! 34 We need to cut down on the number of tests with Heuristics Good tests should have... a high probability of finding faults no duplications be independent test as much code as possible Video 35 Output à 36 Oracle Problem: How do we know that the result is correct? Manually determining if the result is correct is, again, the same unscalable issue we had earlier Test Data Problem: Data is not representative --- some values are over-represented (too many error inputs), others are under-resented (some values completely missing) Tests Completion Problem: How many tests do I need to know that the programme is correct? 1000? 1 Million? All forms of testing have these issues 37 See book: Section 1.7; P13 Black-box testing Examine software function without looking at implementation White-box testing Tests internal structures of implementation Internal perspective of the system is used to create test cases Experience-Based Testing Error guessing / expert opinion Fault insertion Enter faults into code and measure effectiveness of testing 38 See book: Section 1.7.2; P14 Budget viewpoint: Time/Money have expired Activity viewpoint: When software has passed all planned tests Risk management viewpoint: When failure rate meet quality criteria 39 This module focuses on dynamic testing: What is dynamic testing? Code executed, then output is verified Two approaches for static testing: Review-based techniques E.g. pair programming (agile, XP) Code review / walkthrough Program proving Mathematical technique to formally verify correctness Can be done manually / semi-automated / fully automatic (but it is hard and scales poorly!) 40 See book: Section 1.9; P16 1. Unit Testing 2. Integration Testing 3. Regression Testing 4. Sub-system Testing 5. System Testing 6. Acceptance Testing 41 See book: Section 1.10; P17 Real System Actual Results Test Test input Compare Results data Excepted Ideal System Results 42 Actual Results Excepted Results Compare 43 Real System Actual Results Test Test input Compare Results data Excepted Ideal System Results In Java: assertEquals(actual output, expected output) 44 1. Analysis of SW / spec that informs the test 2. Identify test coverage items 3. Identify test cases 4. Verify test design to ensure completeness 5. Implementation of tests 6. Execution of tests 7. Examination of test results 45 For more detail, see book: Section 1.11; P 19 Test Test Log/ Test Cases Test Test Cases Coverage Test Report Cases Items Test Specifications Test Code Test Results Analysis of Test Procedure / Automation Specification / Implementation testing development 46 Section 1.12;P23 Exhaustive Testing is not feasible So, we use heuristics à test (design) techniques E.g. equivalence partitioning (next lecture) Every test technique has a fault model What does this mean? Each test technique is designed to find particular faults (or solves particular issues) – we refer to this as a fault model Initial test techniques have simple fault models Later, they get more sophisticated 47 For more detail, see book: Section 1.13; P24 Technique: Hand Axe Fault Model: Splits wood / moves nails (with limits) Technique: Hammer Fault Model: Moves nails 48 For more detail, see book: Section 1.13 Now you have an idea about: What Software Testing is and why we need it Difference between mistakes, faults, and failures Software development and typical testing activities Testing procedure and artefacts Difference between black-box (BB) and white-box (WB) testing Concept of a fault model Expected vs. actual results in testing 49

Software Testing CS265 Lecture Notes PDF

Document Details

Tags

Related

Summary

Full Transcript

Upgrade to continue