Software Testing Lecture Notes PDF

CPT 441 CSE241/CMM341 Software Quality Foundations Assurance of Software & Testing Engineering Photo credit: Dreamstime.com Topic: Software Testing Learning outcomes Explain t...

CPT 441 CSE241/CMM341 Software Quality Foundations Assurance of Software & Testing Engineering Photo credit: Dreamstime.com Topic: Software Testing Learning outcomes Explain the fundamentals of testing; static vs. dynamic and white-box vs. black Explain box testing. Understand the levels of testing; from testing during development to Understand acceptance testing by system customers. Choose Choose test cases that are geared to discovering program defects Differentiate types of testing – component testing, system testing and release Differentiate testing.  Topic 8: Software Testing 1. SOFTWARE TESTING – THE FUNDAMENTALS 2. DEVELOPMENT TESTING 3. RELEASE TESTING 4. USER TESTING PART 1 SOFTWARE TESTING – THE FUNDAMENTALS  Software testing / Program testing ▪ Testing is intended to show that a program does what it is intended to do and to discover program defects Introduction before it is put into use. ▪ When you test software, you execute a program using artificial data. ▪ You check the results of the test run for errors, anomalies or information about the program’s non- functional attributes. ▪ Can reveal the presence of errors NOT their absence. ▪ Testing is part of a more general verification and validation process, which also includes static validation techniques.  To demonstrate to the developer and the customer that the software meets its requirements. Validation testing For custom software, this means that there should be at least one test for every requirement in the Program requirements document. testing goals For generic software products, it means that there should be tests for all of the system features, plus combinations of these features, that will be incorporated in the product release.  To discover situations in which the behavior of the software is incorrect, undesirable or does not conform to its specification. Program Defect testing is concerned with rooting out undesirable system testing goals behavior such as system crashes, unwanted interactions with other systems, incorrect computations and data corruption. Defect testing Let’s first discuss the terminology and fundamental concepts of software testing TERMINOLOGY – BASED ON ISTQB GLOSSARY Error Failure Fault Defect Bug Software Testing Test cases  Error – A human action that produces an incorrect result. Failure – Deviation of the component/system from its expected delivery, service or result. Defect - a flaw in a component/system that can cause the component/system to fail to perform its required function. A defect, if encountered during execution, may cause a failure of the component/system. e.g. an incorrect statement. Fault, Bug – see defect “Defect” , “Fault” & “Bug” are synonymous terms  LET’S SEE THE CONNECTION: Deviation of the component/system from Fault/Defect its expected delivery, People make service or result. mistakes; they An error made by commit errors a person (e.g. a A failure is the effect of a developer) can defect when executing a result in a defect in program (test object) the software. which appears to the Error “outside” Failure Example: A programmer commits an error by re-using unintended code  WHAT IS SOFTWARE TESTING? Testing - The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements demonstrate that they are fit for purpose detect defects TESTING IS OFTEN REFERRED TO V & V V&V CONFIDENCE Aim of V & V is to establish confidence that the system is ‘fit for purpose’. Software The level of confidence depends on how purpose critical the software is to an organisation. User Users may have low expectations of certain expectations kinds of software. Marketing Getting a product to market early may be more important than finding defects in the environment program.  TEST CASES What are test cases? – A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement How do test cases look like? AN EXAMPLE OF TEST CASES (INPUT VALUES & EXPECTED RESULTS) A program to calculate yearly bonus for employees – based on working years. TEST CASES Input Value 2 4 7 15 (working year) Expected results 500 600 1000 1500 (bonus in RM) Test status Pass Pass Pass Pass  Remarks: ▪ No pre-conditions or post-conditions involved ▪ Only positive tests created  Static Test /Static Verification Concerned with analysis of the static system representation to discover problems Testing of a component/program without execution of the program (can be at requirements or implementation level ) STATIC VS. e.g. reviews or static code analysis DYNAMIC Dynamic Test / Dynamic Verification Concerned with exercising and observing product behaviour Testing that involves the execution of the program Test objects are created during the construction phases The big picture: Test types  STATIC TEST Analysis of a test object (mostly documents) via intensive observation Objective: to detect faults/defects in an artefact – Deviations from standards, requirement defects, design defects, insufficient maintainability and incorrect interface specifications – Check the deviation from project test plans – Results of the analysis can be further used to optimize the development process STATIC TEST Main goal: Defect prevention! –Defects are detected as soon as possible, before they have an effect in the further development and possible required extensive correction and improvements STATIC TESTING: MANUAL VS. AUTOMATED Static testing can be done manually or automatically (tool supports) REVIEWS/INSPECTIONS STATIC ANALYSIS Manual testing by one or more Automated testing by means of people tool support against predefined rules (e.g. coding rules) Human capabilities are used to Used only for documents with a test and evaluate work products formal structure (e.g. source code, UML diagrams). e.g. requirements specs, design specs, code, test plans, test specs, e.g. code compiler checks for test cases, test scripts syntax, static semantics Software Inspections Involve people examining the source representation with the aim of discovering anomalies and defects. Inspections not require execution of a system May applied to any representation of the system (requirements, design, configuration data, test data, etc.). An effective technique for discovering program errors. Reviews/Inspections Advantages of Inspections Incomplete versions of a system can be inspected without additional costs. Check on broader quality attributes of a program, such as compliance with standards, portability and maintainability. Inspections and Testing Inspections and testing are complementary and not opposing verification techniques. Both should be used during the V & V process. Inspections can check conformance with a specification but not conformance with the customer’s real requirements. Inspections cannot check non-functional characteristics such as performance, usability, etc. DYNAMIC TEST Test design techniques Specification- Structure- Experience- based based based Black-box testing White-box testing A model of the software testing process  Uses knowledge of testers, developers, users and other involved with the software, its usage Experience- and its environment based test design Knowledge about likely defects and technique their distribution Black-box vs. white-box testing (1/2) Spot the difference…… Black-box vs. white-box testing (2/2) Spot the difference…… How do we design test cases for black-box and white-box testing? White-box Test design techniques Synonym: Structure-based test design technique Information about how the software is constructed is used to derive the test cases (i.e. the code structure) During the test execution, the inner procedure of the test object is analysed (PoO is inside of the test object) An intervention of the execution is possible, e.g. when observed out of expected (PoC can be inside of the test object) Black-box Test design techniques Synonym: Specification-based test design technique No information available about the program code/internal structure The behaviour of the test object is observed externally (PoO is outside of the test object) Control of the test object is only possible by means of the choice of input test data (PoC is outside of the test object) Models of the test object (requirements, UML models etc) are used for designing test cases  Black-box test design technique (Specification-based) Technique Equivalence Partitioning (EP) Equivalence partitioning (EP) The domain of input and output definitions are divided/partitioned into equivalence classes, such that all values within a class represent a similar behavior of the test object. The assumptions: If a value chosen from the equivalence class (EC) detects a defect, it is assumed that this defect will also be detected by every other value in the EC vice versa EC 1 EC 2 EC 3 EC 4 Domain of definition EP Procedure Determine input and output domains from the specification Perform EP for each of the following: If an area of validity is specified, then, define one valid and two invalid ECs. If a min and a max value are specified, then define one valid and two invalid ECs. If a number of values are specified which are not treated separately, then, define one valid EC and in addition define one invalid EC If a “true” and “false” situations are specified, then, define ECs for the true and false conditions On a last step, one representative of each EC is determined as well as the result to be expected for it (i.e. the test cases) An example A program that receives an input integer with possible values from 1 to 100 If an area of validity is specified, then, define one valid and two invalid ECs. Invalid EC Valid EC Invalid EC (input value 1 – 100) EP Coverage Let’s try some examples… Example 1 Given a specification of a program as below: A program that accepts an integer value from 1 to 100. Create effective test cases using the Equivalence Partitioning (EP) technique for testing this requirement specification. Example 1 – the steps A program that accepts an integer value from 1 to 100. If an area of validity is specified, then, define one valid and two invalid ECs. Let’s assign x as the input value; 0 ≤ x ≤ 100 Then we can initially identify three equivalence classes: 1. Invalid EC x 100 100  IF Value >= 0 AND Value