Fundamentals of Software Testing 2nd Edition PDF

Table of Contents Cover Table of Contents Title Page Copyright Page Preface Glossary 1 Fundamentals of Testing 1.1. What is testing? 1.2. What is testing? 1.3. Paradoxes and main principles 1.4. Test activities, testware and test roles 1.5. Role...

Table of Contents Cover Table of Contents Title Page Copyright Page Preface Glossary 1 Fundamentals of Testing 1.1. What is testing? 1.2. What is testing? 1.3. Paradoxes and main principles 1.4. Test activities, testware and test roles 1.5. Roles in testing 1.6. Essential skills and “good practices” in testing 1.7. Testers and code of ethics (FL 1.6) 1.8. Sample exam questions 2 Testing Throughout the Software Life Cycle 2.1. Testing through the software development life cycle 2.2. Test levels and test types 2.3. Types of tests 2.4. Test and maintenance 2.5. Oracles 2.6. Process improvements 2.7. Specific cases 2.8. Sample exam questions 3 Static Testing 3.1. Static techniques and the test process  3.2. Review process 3.3. Static analysis by tools 3.4. Added value of static activities 3.5. Sample exam questions 4 Test Design Techniques 4.1. The test development process 4.2. Categories of test design techniques 4.3. Black-box techniques 4.4. Structure-based techniques 4.5. Experience-based technique 4.6. Collaboration-based test approaches 4.7. Choosing test techniques 4.8. Sample exam questions 5 Test Management 5.1. Test organization 5.2. Test planning and estimation 5.3. Test progress monitoring and control (FL 5.3) 5.4. Reporting 5.5. Transverse processes and activities 5.6. Risk management (FL 5.2) 5.7. Defect management (FL 5.5) 5.8. Sample exam questions 6 Tools Support for Testing 6.1. Types of test tools 6.2. Assumptions and limitations of test tools 6.3. Selecting and introducing tools in an organization 6.4. Sample exam questions 7 Mock Exam 8 Templates and Models  8.1. Master test plan 8.2. Test plan 8.3. Test design document 8.4. Test case 8.5. Test procedure 8.6. Test log 8.7. Defect report 8.8. Test report 9 Answers to the Questions 9.1. Answers to the end-of-chapter questions 9.2. Correct answers to the sample paper questions References Index Other titles from ISTE in Computer Engineering End User License Agreement List of Tables Chapter 2 Table 2.1. Development cycle comparison Chapter 3 Table 3.1. Comparison of review types Table 3.2. Defects in data flow analysis Chapter 4 Table 4.1. Valid and invalid equivalence partitions Table 4.2. Expanded decision table Table 4.3. Reduced decision table Table 4.4. State transition table representation  Chapter 5 Table 5.1. Risk likelihood Table 5.2. Impact severity of the risks Table 5.3. Risks by severity and likelihood List of Illustrations Chapter 1 Figure 1.1. Origin and impacts of defects Figure 1.2. Fundamental test processes Chapter 2 Figure 2.1. Waterfall model Figure 2.2. V-model Figure 2.3. W-model. Figure 2.4. Iterative model Figure 2.5. Spiral model Figure 2.6. Incremental model Figure 2.7. Scrum development model Figure 2.8. ISO 25010 quality characteristics Figure 2.9. Example of transaction budget. Figure 2.10. Impact analysis. Figure 2.11. Bathtub curve Chapter 3 Figure 3.1. Static and dynamic techniques and defects Figure 3.2. Types of objectives per level of review Figure 3.3. Static analysis graph Figure 3.4. Optimized website architecture.  Figure 3.5. Architecture that is not optimized. Figure 3.6. Control flow Figure 3.7. Control flow and data flow Figure 3.8. Example of data flow for three variables. Chapter 4 Figure 4.1. Traceability and coverage follow-up Figure 4.2. Test techniques Figure 4.3. Example of fields. Figure 4.4. Expense reimbursement for kilometers traveled in France in 2010 fo... Figure 4.5. State transition diagram for a telephone Figure 4.6. State transition diagram Figure 4.7. Representation of a decision Figure 4.8. Representation of instructions Figure 4.9. Code of function “Factorial”. Figure 4.10. Control flow for function “factorial”. Figure 4.11. Instruction coverage Figure 4.12. Control flow – horizontal representation Figure 4.13. Example of alternatives Figure 4.14. Linked conditions Figure 4.15. Control flow for linked conditions Figure 4.16. Branch coverage Figure 4.17. Control flow instruction “case of” Figure 4.18. Control flow with two conditions. Figure 4.19. Boolean decision table Figure 4.20. Decision and condition coverage. Chapter 5 Figure 5.1. Uniform test distribution. Figure 5.2. Non-uniform test distribution. Figure 5.3. Documentary architecture IEEE 829 v1998 and v2008. Figure 5.4. Example of integration. Figure 5.5. Defect detection and correction ((♦) opened; (△) closed).... Figure 5.6. Quality of a milestone Figure 5.7. Quality indices of the reviews. Figure 5.8. Dashboard. Figure 5.9. Time-over-time diagram. Figure 5.10. Table of risks: occurrence × impact Figure 5.11. Evolution of the risks throughout time  Revised and Updated 2nd Edition Fundamentals of Software Testing Bernard Homès First edition published 2011 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc., © ISTE Ltd 2011. This edition published 2024 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address: ISTE Ltd 27-37 St George’s Road London SW19 4EU UK www.iste.co.uk John Wiley & Sons, Inc. 111 River Street Hoboken, NJ 07030 USA www.wiley.com © ISTE Ltd 2024 The rights of Bernard Homès to be identified as the author of this work have been asserted by him in accordance with the Copyright, Designs and Patents Act 1988. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s), contributor(s) or editor(s) and do not necessarily reflect the views of ISTE Group. Library of Congress Control Number: 2024932622 British Library Cataloguing-in-Publication Data A CIP record for this book is available from the British Library ISBN 978-1-78630-982-2 Preface “My eldest brother sees the spirit of sickness and removes it before it takes shape, so his name does not get out of the house. My elder brother cures sickness when it is still extremely minute, so his name does not get out of the neighborhood. As for me, I puncture veins, prescribe potions, and massage skin, so from time to time my name gets out and is heard among the lords”. – Sun Tzu, The Art of War I often turn to the above quote, replacing “sickness” by “defects”, and applying it to software instead of humans. I have seen few eldest brothers, a number of elder ones, and am perhaps in the last category of practitioners. Why this book? As we know, software testing is increasingly important in the industry, reflecting the increasing importance of software quality in today’s world. Since 2011, when the first edition of this book was published, there have been evolutions in the software industry that have impacted software testing. This new – revised – edition will help testers adopt more up-to-date fundamental practices. Due to the lack of formal and recognized training in software testing, a group of specialist consultants gathered together in 2002 and founded the International Software Testing Qualifications Board (ISTQB). They defined the minimal set of methodological and technical knowledge that testers should know depending on their experience. This was gathered into what is called a syllabus. The foundation level syllabus was reviewed in 2023 and has been the basis of an international certification scheme that has already been obtained by more than 1,000,000 testers worldwide. This book can serve as reference material for testers preparing the ISTQB foundation level exam, and for any beginner testers. It references the 2023 version of the ISTQB Certified Tester Foundation Level syllabus. This book follows the order and chapters of the syllabus, which should help you to successfully complete the certification exam. It is a one-stop reference book offering you: more detailed explanations than those found in the ISTQB syllabus; definitions of the terms (i.e. the Glossary) used in the certification exams; practice questions similar to those encountered during the certification exam; a sample exam. For testers who want to acquire a good understanding of software and system tests, this book provides the fundamental principles as described by the ISTQB and recognized experts. This book provides answers and areas of discussion to enable test leaders and managers to: improve their understanding of testing; have an overview of process improvement linked to software testing; increase the efficiency of their software development and tests. Throughout this book, you will find learning objectives (denoted as FL-…) that represent the ISTQB foundation level syllabus learning objectives. These are the topics that certification candidates should know and that are examined in certification exams. Prerequisite Software testing does not require specific prerequisites, but a basic understanding of data processing and software will allow you to better understand software testing. The reader with software development knowledge, whatever the programming language, will understand certain aspects faster, but a simple practice as a user should be enough to understand this book. ISTQB and national boards The ISTQB is a not-for-profit international association grouping national software testing boards covering over 50 countries. These national boards are made up of software testing specialists, consultants and experts, and together they define the syllabi and examination directives for system and software testers. A number of prominent authors of software testing books participated in the creation of the initial syllabi, ensuring that they reflect what a tester should know depending on their level of experience (foundation, advanced, expert) and their objectives (test management, functional testing and test techniques, specialization in software security or performance testing, etc.). Glossary, syllabus and business outcomes The ISTQB is aware of the broad diversity of terms used and the associated diversity of interpretation of these terms depending on the customers, countries and organizations. A common glossary of software testing terms has been set up and national boards provide translation of these terms in national languages to promote better understanding of the terms and the associated concepts. This becomes increasingly important in a context of international cooperation and offshore sub-contracting. The syllabi define the basis of the certification exams; they also help to define the scope of training and are applicable at three levels of experience: foundation level, advanced level and expert level. This book focuses on the foundation level. The foundation level, made up of a single module, is detailed in the following chapters. Expected business outcomes, as stated by the ISTQB, are as follows for foundation level testers: understand what testing is and why it is beneficial; understand the fundamental concepts of software testing;  identify the test approach and activities to be implemented depending on the context of testing; assess and improve the quality of documentation; increase the effectiveness and efficiency of testing; align the test process with the software development life cycle; understand test management principles; write and communicate clear and understandable defect reports; understand the factors that influence the priorities and efforts related to testing; work as part of a cross-functional team; know the risks and benefits related to test automation; identify the essential skills required for testing; understand the impact of risk on testing; effectively report on test progress and quality. As we can see, the work of a tester impacts many different aspects in software development, from evaluating the quality of input documentation (specifications, requirements, user stories, etc.) to reporting on progress and risks, to test automation and interacting with the development teams to understand what to test and explain what defects are identified. ISTQB certification The ISTQB proposes software tester certifications, which are recognized as equivalent by all ISTQB member boards throughout the world. The level of difficulty of the questions and the exams are based on identical criteria (defined in the syllabi) and terminology (defined in the Glossary). The certification exams proposed by the ISTQB enable the candidates to validate their knowledge, and assure employers or potential customers of a minimum level of knowledge from their testers, whatever their origin. Training providers deliver courses to help participants succeed in the certification exams, however much of the training involves brain cramming sessions and does not ensure that the participant has the required level of autonomy to succeed in the profession. This book attempts to identify the necessary skills, as well as provide the reader with a concentrate of more than 40 years of practice in the field of software quality and testing. The ISTQB certifications are recognized as equivalent throughout the whole world, enabling international cross-recognition. Key for understanding the content To be used efficiently, this book has the following characteristics: FL-xxx: text that starts with FL-xxx is a reminder of the learning objectives present in the ISTQB foundation level syllabus for certified testers. Those objectives are expanded in the paragraphs following this tag. The titles of the chapters correspond to those of the ISTQB foundation level syllabus, version 2011. This is also often the case for the section heads; the syllabus reference is provided in the form (FLx.y), where x.y stands for the chapter and section head of the ISTQB foundation level syllabus. A synopsis closes each of the chapters, summarizing the aspects covered and identifying the terms in the glossary that should be known for the certification exam. Sample exam questions are also provided at the end of each chapter. These questions were developed by applying the same criteria as for the creation of real exam questions. The sample questions provided in Chapters 1–6 have been reproduced with the kind permission of © Bernard Homès 2011. March 2024 Glossary The definitions listed below have been extracted from the International Software Testing Qualifications Board (ISTQB) Standard Glossary of Terms used in Software Testing. Only the terms used for the Foundation Level certification exams are mentioned, so as not to drown the reader in terms that are used at other levels or in other syllabi. Acceptance criteria: The criteria that a component or system must satisfy in order to be accepted by a user, customer or other authorized entity (from ISO 24765). Acceptance test-driven development: A collaboration-based test-first approach that defines acceptance tests in the stakeholders’ domain language. Abbreviation: ATDD. Acceptance testing: Formal testing with respect to user needs, requirements and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entities to determine whether or not to accept the system. See also user acceptance testing. ACM: (Association for Computer Machinery) professional and scientific association for the development of information technology. Alpha testing: Simulated or actual operational testing by potential users/customers or an independent test team at the developers’ site, but outside the development organization. Alpha testing is often employed as a form of internal acceptance testing. Anomaly: A condition that deviates from expectation (from ISO 24765). Attack: Directed and focused attempt to evaluate the quality, and especially the reliability, of a test object by attempting to force specific failures to occur. Beta testing: Operational testing by potential and/or existing users/customers at an external site not otherwise involved with the developers, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes. Beta testing is often employed as a form of external acceptance testing in order to acquire feedback from the market. Black-box technique: See also black-box testing. Black-box test technique: A test technique based on an analysis of the specification of a component or system. Synonyms: black-box test design technique, specification-based test technique. Black-box testing: Testing, either functional or nonfunctional, based on an analysis of the specification of the component or system. Synonym: specification-based testing. Boundary value analysis: A black-box test technique in which test cases are designed based on boundary values. See also boundary values. Branch coverage: The coverage of branches in a control flow graph (percentage of branches that have been exercised by a test suite). One hundred percent branch coverage implies both 100% decision coverage and 100% statement coverage. Bug: See also defect. Checklist-based testing: An experience-based test technique in which test cases are designed to exercise the items of a checklist. Code coverage: An analysis method that determines which parts of the software have been executed (covered) by the test suite and which parts have not been executed, for example, statement coverage, decision coverage or condition coverage. Collaboration-based test approach: An approach to testing that focuses on defect avoidance by collaborating among stakeholders. Commercial off-the-shelf software (COTS): See also off-the-shelf software. Compiler: A software tool that translates programs expressed in a high- order language into their machine language equivalents. Complexity: The degree to which a component or system has a design and/or internal structure that is difficult to understand, maintain and verify. See also cyclomatic complexity. Component integration testing: The testing executed to identify defects in the interfaces and interactions between integrated components. Synonyms: module integration testing, unit integration testing. Component testing: A test level that focuses on individual hardware or software components. Synonyms: module testing, unit testing. Configuration control: An element of configuration management, consisting of the evaluation, coordination, approval or disapproval, and implementation of changes to configuration items after formal establishment of their configuration identification. Configuration item: An aggregation of hardware, software or both, that is designated for configuration management and treated as a single entity in the configuration management process. Configuration management: A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specified requirements. Confirmation testing: A type of change-related testing performed after fixing a defect to confirm that a failure caused by that defect does not reoccur. Synonym: re-testing. Control flow: An abstract representation of all possible sequences of events (paths) in the execution through a component or system. Coverage: The degree to which specified coverage items are exercised by a test suite, expressed as a percentage. Synonym: test coverage. Coverage item: An attribute or combination of attributes derived from one or more test conditions by using a test technique. See also coverage criteria. Coverage measurement tool: See also coverage tool. Coverage tool: A tool that provides objective measures of what structural elements, for example, statements, branches, have been exercised by the test suite. Cyclomatic complexity: The number of independent paths through a program. Cyclomatic complexity is defined as: L – N + 2P, where: L = the number of edges/links in a graph; N = the number of nodes in a graph; P = the number of disconnected parts of the graph (e.g. a calling graph and a subroutine). Data-driven testing: A scripting technique that stores test input and expected results in a table or spreadsheet, so that a single control script can execute all of the tests in the table. Data-driven testing is often used to support the application of test execution tools such as capture/playback tools. See also keyword-driven testing. Data flow: An abstract representation of the sequence and possible changes of the state of data objects, where the state of an object can be creation, usage or destruction. Debugging: The process of finding, analyzing and removing the causes of failures in a component or system. Debugging tool: A tool used by programmers to reproduce failures, investigate the state of programs and find the corresponding defect. Debuggers enable programmers to execute programs step-by-step, halt a program at any program statement and set and examine program variables. Decision coverage: The percentage of decision outcomes that have been exercised by a test suite. One hundred percent decision coverage implies both 100% branch coverage and 100% statement coverage. Decision table testing: A black-box test technique in which test cases are designed to exercise the combinations of conditions inputs and/or stimuli (causes) shown in a decision table. Defect: An imperfection or deficiency in a work product, which can cause the component or system to fail to perform its required function, for example, an incorrect statement or data definition (from ISO 24765). A defect, if encountered during execution, may cause a failure of the component or system. Synonyms: bug, fault. Defect density: The number of defects identified in a component or system divided by the size of the component or system (expressed in standard measurement terms, for example, lines-of-code, number of classes or function points). Defect management: The process of recognizing, recording, classifying, investigating, resolving and disposing of defects. It involves recording defects, classifying them and identifying the impact. Defect management tool: See also incident management tool. Defect report: Documentation of the occurrence, nature and status of a defect. Synonym: bug report. Driver: A software component or test tool that replaces a component that takes care of the control and/or the calling of a component or system. Dynamic analysis tool: A tool that provides run-time information on the state of the software code. These tools are most commonly used to identify unassigned pointers, check pointer arithmetic, and monitor the allocation, use and de-allocation of memory and highlight memory leaks. Dynamic testing: Testing that involves the execution of the test item/software of a component or system (from ISO 29119-1). See also static testing. Entry criteria: The set of generic and specific conditions that permit a process to proceed with a defined task (from Gilb and Graham), for example, test phase. The purpose of entry criteria is to prevent a task that would entail more (wasted) effort compared to the effort needed to remove the failed entry criteria from starting. See also exit criteria. Equivalence partitioning: A black-box test technique in which test conditions are equivalence partitions exercised by one representative member of each partition (from ISO 29119-1). Synonym: partition testing. Error: A human action that produces an incorrect result (from ISO 24765). Synonym: mistake. Error guessing: A test design technique in which tests are derived on the basis of the tester’s knowledge of past failures, or general knowledge of failure modes, in order to anticipate the defects that may be present in the component or system under test as a result of errors made, and design tests specifically to expose them (from ISO 29119-1). Exhaustive testing: A test approach in which the test suite comprises all combinations of input values and preconditions. Exit criteria: The set of generic and specific conditions, agreed upon with the stakeholders, that permit a process to be officially completed. The purpose of exit criteria is to prevent a task from being considered completed when there are still outstanding parts of the task that have not been finished. Exit criteria are used by testing to report against and plan when to stop testing (after Gilb and Graham). Synonyms: test completion criteria, completion criteria. See also entry criteria. Experience-based test technique: A test technique based on the tester’s experience, knowledge and intuition. Synonyms: experience-based test design technique, experience-based technique. Exploratory testing: An approach to testing in which the testers dynamically design and execute tests based on their knowledge, exploration of the test item and the results of previous tests. This is used to design new and better tests (from ISO 29119-1). See also test charter. Failure: An event in which a component or system does not perform a required function within specified limits (from ISO 24765). Actual deviation of the component or system from its expected delivery, service or result (according to Fenton). The inability of a system or system component to perform a required function within specified limits. A failure may be produced when a fault is encountered [EUR 00]. Failure rate: The ratio of the number of failures of a given category to a given unit of measure, for example, failures per unit of time, failures per number of transactions and failures per number of computer runs. Fault attack: See also attack. Field testing: See also beta testing. Finite state testing: See also state transition testing. Formal review: A type of review that follows a defined process with a formally documented output, for example, inspection (from ISO 20246). Functional requirement: A requirement that specifies a function that a component or system must perform. Functional testing: Testing performed to evaluate if a component or system satisfies functional requirements (from ISO 24765). See also black-box testing. Horizontal traceability: The tracing of requirements for a test level through the layers of test documentation (e.g. test plan, test design specification, test case specification and test procedure specification). IEEE: Institute for Electrical and Electronic Engineers, a professional, not- forprofit association for the advancement of technology, based on electrical and electronic technologies. This association is active in the design of standards. There is a French chapter on this association, which provides publications that are useful for software testers. Impact analysis: The assessment of change to the layers of development documentation, test documentation and components, in order to implement a given change to specified requirements. Incident: Any event occurring during testing which requires investigation. Incident management tool: A tool that facilitates the recording and status tracking of incidents found during testing. They often have workflow- oriented facilities to track and control the allocation, correction and re- testing of incidents and provide reporting facilities. See also defect management tool. Incident report: A document reporting on any event that occurs during the testing which requires investigation. Incremental development model: A development life cycle where a project is broken into a series of increments, each of which delivers a portion of the functionality in the overall project requirements. The requirements are prioritized and delivered in priority order in the appropriate increment. In some (but not all) versions of this life cycle model, each sub-project follows a “mini V-model” with its own design, coding and testing phases. Independence of testing: Separation of responsibilities, which encourages the accomplishment of objective testing. Informal review: A type of review that does not follow a defined process and has no formally documented output. Inspection: A type of formal review that relies on visual examination of documents to detect defects, for example, violations of development standards and non-conformance to higher-level documentation, and uses defined team roles and measurements to identify defects in a work product and improve the review and software development processes. The most formal review technique and, therefore, always based on a documented procedure (from ISO 20246). See also peer review. Intake test: A special instance of a smoke test to decide whether the component or system is ready for detailed and further testing. An intake test is typically carried out at the start of the test execution phase. See also smoke test. Integration: The process of combining components or systems into larger assemblies. Integration testing: Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. See also component integration testing, system integration testing. Interoperability testing: The process of testing to determine the interoperability of a software product. ISTQB: International Software Testing Qualifications Board, a nonprofit association developing international certification for software testers. Keyword-driven testing: A scripting technique that uses data files to contain not only test data and expected results, but also keywords related to the application being tested. The keywords are interpreted by special supporting scripts that are called by the control script for the test. See also data-driven testing. Maintainability testing: The process of testing to determine the maintainability of a software product. Maintenance testing: Testing the changes to an operational system or the impact of a changed environment on an operational system. Master test plan: See also project test plan. Metric: A measurement scale and the method used for measurement. Mistake: See also error. Modeling tool: A tool that supports the validation of models of the software or system. Moderator: The leader and main person responsible for an inspection or other review process. Non-functional testing: Testing performed to evaluate whether a component or system complies with nonfunctional requirements. N-switch coverage: The percentage of sequences of N+1 transitions that have been exercised by a test suite. N-switch testing: A form of state transition testing in which test cases are designed to execute all valid sequences of N+1 transitions (Chow). See also state transition testing. Off-the-shelf software: A software product that is developed for the general market, that is, for a large number of customers, and that is delivered to many customers in identical format. Oracle: See also test oracle. Peer review: See also technical review. Performance testing: The process of testing to determine the performance of a software product. Performance testing tool: A tool to support performance testing, that usually has two main facilities: load generation and test transaction measurement. Load generation can simulate either multiple users or high volumes of input data. During execution, response time measurements are taken from selected transactions and these are logged. Performance testing tools normally provide reports based on test logs and graphs of load against response times. Portability testing: The process of testing to determine the portability of a software product. Probe effect: The effect on the component or system when it is being measured, for example, by a performance testing tool or monitor. For example, performance may be slightly worse when performance testing tools are being used. Product risk: A risk impacting the quality of a product and directly related to the test object. See also risk. Project risk: A risk related to management and control of the (test) project, for example, lack of staffing, strict deadlines, changing requirements, etc., that impacts project success. See also risk. Project test plan: A test plan that typically addresses multiple test levels. See also master test plan. Quality: The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations (from IREB). Quality assurance: Activities focused on providing confidence that quality requirements will be fulfilled. Abbreviation: QA (from ISO 24765). See also quality management. RAD: Rapid Application Development, a software development model. Regression testing: A type of change-related testing to detect whether defects have been introduced or uncovered in unchanged areas of the software. It is performed when the software or its environment is changed. Reliability testing: The process of testing to determine the reliability of a software product. Requirement: A condition or capability needed by a user to solve a problem or achieve an objective that must be met or possessed by a system or system component to satisfy a contract, standard, specification or other formally imposed document. Requirement management tool: A tool that supports the recording of requirements, attributes of requirements (e.g. priority, knowledge responsible), and annotation, and facilitates traceability through layers of requirements and requirement change management. Some requirement management tools also provide facilities for static analysis, such as consistency checking and violations to pre-defined requirement rules. Re-testing: Testing that runs test cases that failed the last time they were run, in order to verify the success of corrective actions. Review: A type of static testing in which a work product or process is evaluated by one or more individuals to detect defects or provide improvements. Examples include management review, informal review, technical review, inspection and walk-through. Review tool: A tool that provides support to the review process. Typical features include review planning, tracking support, communication support, collaborative reviews and a repository for collecting and reporting of metrics. Reviewer: The person involved in the review who identifies and describes anomalies in the product or project under review. Reviewers can be chosen to represent different viewpoints and roles in the review process. Risk: A factor that could result in future negative consequences; usually expressed as impact and likelihood. See also product risk, project risk. Risk analysis: The overall process of risk identification and risk assessment. Risk assessment: The process to examine identified risks and determine the risk level. Risk-based testing: Testing in which the management, selection, prioritization and use of testing activities and resources are based on corresponding risk types and risk levels. This approach is used to reduce the level of product risks and inform stakeholders about their status, starting in the initial stages of a project (from ISO 29119-1). Risk control: The overall process of risk mitigation and risk monitoring. Risk identification: The process of finding, recognizing and describing risks. (from ISO 31000). Risk level: The measure of a risk defined by risk impact and risk likelihood. Synonym: risk exposure. Risk management: The process for handling risks (from ISO 24765). Risk mitigation: The process through which decisions are reached and protective measures are implemented to reduce risks or maintain them at specified levels. Risk monitoring: The activity that checks and reports the status of known risks to stakeholders. Robustness testing: Testing to determine the robustness of the software product. Root cause: A source of a defect such that if it is removed, the occurrence of the defect type is decreased or removed (from CMMI). SBTM: Session-based test management, an ad hoc and exploratory test management technique, based on fixed length sessions (from 30 to 120 minutes), during which testers explore a part of the software application. Scribe: The person who has to record each defect mentioned and any suggestions for improvement during a review meeting, on a logging form. The scribe has to ensure that the logging form is readable and understandable. Scripting language: A programming language in which executable test scripts are written, used by a test execution tool (e.g. a capture/replay tool). Security testing: Testing to determine the security of the software product. Shift left: An approach to perform testing and quality assurance activities as early as possible in the software development life cycle. Site acceptance testing: Acceptance testing by users/customers on site, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes, normally including hardware as well as software. SLA: Service-level agreement, service agreement between a supplier and their client, defining the level of service a customer can expect from the provider. Smoke test: A subset of all defined/planned test cases that cover the main functionality of a component or system, to ascertain that the most crucial functions of a program work, but not bothering with finer details. A daily build and smoke test is among industry best practices. State transition: A transition between two states of a component or system. State transition testing: A black-box test design technique in which test cases are designed to exercise elements of a state transition model, and execute valid and invalid state transitions. Synonym: finite state testing. See also N-switch testing. Statement coverage: The percentage of executable statements that have been exercised by a test suite. Static analysis: The process of evaluating a component or system without executing it, based on its form, structure, content or documentation, for example, requirements or code, carried out without execution of these software artifacts (from ISO 24765). Static code analyzer: A tool that carries out static code analysis. The tool checks the source code for certain properties, such as conformance to coding standards, quality metrics or data flow anomalies. Static testing: Testing of a component or system at the specification or implementation level without execution of that software, for example, reviews or static code analysis. See also dynamic testing. Stress testing: A type of performance testing conducted to evaluate a system or component at or beyond the limits of its anticipated or specified workloads, or with reduced availability of resources, such as access to memory or servers. See also performance testing, load testing. Stress testing tool: A tool that supports stress testing. Structural testing: See also white-box testing. Stub: A skeletal or special-purpose implementation of a software component, used to develop or test a component that calls or is otherwise dependent on it. It replaces a called component. System integration testing: Testing the integration of systems and packages; testing the interfaces of external organizations (e.g. electronic data interchange, the Internet). System testing: A test level that focuses on verifying that a system as a whole meets specified requirements. Technical review: A formal peer group discussion/review by technical experts who examine the quality of a work product and identify discrepancies from specifications and standards. See also peer review. Test: A set of one or more test cases. Test analysis: The activity that identifies test conditions by analyzing the test basis. Test approach: The implementation of the test strategy for a specific project. It typically includes the decisions made that follow based on the (test) project’s goal and the risk assessment carried out, starting points regarding the test process, the test design techniques to be applied, exit criteria and test types to be performed. Test automation: The use of software to perform or support test activities. Test basis: All documents from which the requirements of a component or system can be inferred. The documentation on which the test cases are based. If a document can be amended only by way of a formal amendment procedure, then the test basis is called a frozen test basis (from TMap). Test case: A set of execution preconditions, input values, actions (where applicable), expected results and post-conditions, developed based on test conditions, such as to exercise a particular program path or verify compliance with a specific requirement. See also test step. Test case specification: A document specifying a set of test cases (objective, inputs, test actions, expected results and execution preconditions) for a test item. Test comparator: A test tool to perform automated test comparison. Test completion: The activity that makes testware available for later use, leaves the test environments in a satisfactory condition and communicates the results of testing to relevant stakeholders. Test completion report: A type of test report produced at completion milestones that provides an evaluation of the corresponding test items against exit criteria. Synonym: test summary report. Test condition: A testable aspect of a component or system identified that could be verified by one or more test cases, for example, a function, transaction, quality attribute or structural element (from ISO 29119-1). Synonyms: test situation, test requirement. Test control: A test management task that deals with developing and applying a set of corrective actions to get a test project on track when monitoring shows a deviation from what was planned. See also test management. Test coverage: See also coverage. Test data: Data that exists/is needed (e.g. in a database) before a test is executed, and that affects or is affected by the component or system under test. Synonym: test dataset. Test data preparation tool: A type of test tool that enables data to be selected from existing databases or created, generated, manipulated and edited for use in testing. Test design: The process of transforming general testing objectives into tangible test conditions and test cases. See also test design specification. Test design specification: A document that specifies the test conditions (coverage items) for a test item, the detailed test approach, and identifies the associated high-level test cases. Test design technique: A method used to derive or select test cases. Test design tool: A tool that supports the test design activity by generating test inputs from a specification that may be held in a CASE tool repository, for example, the requirements management tool, or from specified test conditions held in the tool itself. Test-driven development: Agile development method, where the tests are designed and automated before the code (from the requirements or specifications), then a minimal amount of code is written to successfully pass the test. This iterative method ensures that the code continues to fulfill requirements via test execution. Test environment: An environment containing hardware, instrumentation, simulators, software tools and other support elements needed to conduct a test. Test execution: The process of running a test using the component or system under test, producing actual results. Test execution schedule: A scheme for the execution of test procedures. The test procedures are included in the test execution schedule in their context and in the order in which they are to be executed. Test execution tool: A type of test tool that is able to execute other software using an automated test script, for example, capture/playback. Test harness: A test environment composed of stubs and drivers needed to conduct a test. Test implementation: The activity that prepares the testware needed for test execution based on test analysis and design. Test leader: See also test manager. Test level: A group of test activities that are organized and managed together. A test level is linked to the responsibilities in a project. Examples of test levels: component test, integration test, system test and acceptance test. Synonym: test stage. Test log: A chronological record of relevant details about the execution of tests. Test management: The planning, estimating, monitoring and control of test activities, typically carried out by a test manager. Test manager: The person responsible for testing and evaluating a test object. The individual who directs, controls, administers, plans and regulates the evaluation of a test object. Test monitoring: A test management task that deals with the activities related to periodically checking the status of a test project. Reports are prepared that compare the results with what was expected. See also test management. Test object: The work product to be tested. Synonym: test item. Test objective: A reason or purpose for designing and executing a test. Synonym: test goal. Test oracle: A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), a user manual or an individual’s specialized knowledge, but should not be the code (from Adrion). Synonym: oracle. Test plan: A document describing the scope, approach, resources and schedule of intended test activities. Among others, it identifies test items, the features to be tested, the testing tasks, who will do each task, the degree of tester independence, the test environment, the test design techniques, the test measurement techniques to be used, the rationale for their choice and any risks requiring contingency planning. It is a record of the test planning process (from ISO 29119-1). See also master test plan, level test plan, test scope. Test planning: The activity of establishing or updating a test plan. Test policy: A high-level document describing the principles, approach and major objectives of the organization regarding testing. Test procedure: A sequence of test cases in execution order, and any associated actions that may be required to set up the initial preconditions and any wrap-up activities post execution (from ISO 29119-1). See also test procedure specification. Test procedure specification: A document specifying a sequence of actions for the execution of a test; also known as the test script or manual test script. Test process: The set of interrelated activities comprised of test planning, test monitoring and control, test analysis, test design, test implementation, test execution and test completion. Test progress report: A type of periodic test report that includes the progress of test activities against a baseline, risks and alternatives requiring a decision. Synonym: test status report. Test pyramid: A graphical model representing the amount of testing per level, with more at the bottom than at the top. Test report: See also test summary report. Test result: The consequence/outcome of the execution of a test. Synonyms: outcome, test outcome, result. Test script: Commonly used to refer to a test procedure specification, especially an automated one. Test strategy: A high-level document defining the test levels to be performed and the testing within those levels for a program (one or more projects). Test suite: A set of several test cases for a component or system under test, where the post-condition of one test is often used as the precondition for the next one. Test summary report: A document summarizing testing activities and results. It also contains an evaluation of the corresponding test items against exit criteria. Test technique: A procedure used to define test conditions, design test cases and specify test data. Synonym: test design technique. Test type: A group of test activities based on specific test objectives aimed at specific characteristics of a component or system (from TMap). Tester: A technically skilled professional who is involved in the testing of a component or system. Testing: The process within the software development life cycle that evaluates the quality of a component or system and related work products. See also quality control. Testing quadrants: A classification model of test types/test levels in four quadrants, relating them to two dimensions of test objectives: supporting the product team versus critiquing the product, and technology-facing versus business-facing. Testware: Artifacts produced during the test process required to plan, design and execute tests, such as documentation, scripts, inputs, expected results, setup and clear-up procedures, files, databases, environment and any additional software or utilities used in testing (from ISO 29119-1). Thread testing: A version of component integration testing where the progressive integration of components follows the implementation of subsets of the requirements, as opposed to the integration of components by levels of a hierarchy. Traceability: The ability to identify related items in documentation and software, such as requirements with associated tests (from IREB). See also horizontal traceability, vertical traceability. Usability testing: Testing to determine the extent to which the software product is understood, easy to learn, easy to operate and attractive to the users under specified conditions. Use case testing: A black-box test design technique in which test cases are designed to execute user scenarios. User acceptance testing: See also acceptance testing. Validation: Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled (from IREB). Verification: Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled (from ISO 9000). Version control: See also configuration control. Vertical traceability: The tracing of requirements through the layers of development documentation to components. V-model: A framework to describe the software development life cycle activities from requirement specification to maintenance. The V-model illustrates how testing activities can be integrated into each phase of the software development life cycle. Walkthrough: A step-by-step presentation by the author of a document in order to gather information and establish a common understanding of its content. This may lead members of the review to ask questions and make comments about possible issues (from ISO 20246). Synonym: structured walkthrough. See also peer review. White-box test technique: A test technique only based on the internal structure of a component or system. Synonyms: white-box test design technique, structure-based test technique. White-box testing: Testing based on an analysis of the internal structure of the component or system. Synonyms: clear-box testing, code-based testing, glass-box testing, logic-coverage testing, logic-driven testing, structural testing, structure-based testing. 1 Fundamentals of Testing 1.1. What is testing? In our everyday life, we are dependent on the correct execution of software, whether it is in our equipment (cell phones, washing machine, car engine injection, etc.), in the transactions we undertake each day (credit or debit card purchases, fund transfers, Internet usage, electronic mail, etc.), or even those that are hidden from view (back office software for transaction processing); software simplifies our daily lives. When it goes awry, the impact can be devastating. Some defects are pretty innocuous, and some may only show themselves under very unusual circumstances. However, some of those defects can have very serious consequences, and some of those defects can lead to serious failures. In some cases, the effect of the failures is financial loss or inconvenience. In some cases, physical consequences, such as injury or death, can occur. When you hear the word “testing”, you may think of someone sitting in front of a computer, entering information, observing the results and reporting defects; i.e. executing tests. That is certainly part of testing, and often the most visible to outside stakeholders. However, as we will see later, there are multiple activities associated with testing, some focused on avoiding introduction of defects and some focused on identifying – and later removing – defects. 1.1.1. Software and systems context Testing software and systems is necessary to avoid failures visible to customers and avoid bad publicity for the organizations involved. This is the case for service companies responsible for the development or testing of third-party software, because the customer might not renew the contract, or might sue for damages. We can imagine how millions of Germans felt on January 1, 2010, when their credit cards failed to work properly. There was no early warning, and they found themselves, the day after New Year celebrations, with an empty fridge, totally destitute, without the possibility of withdrawing cash from ATMs or purchasing anything from retail outlets. Those most pitied were probably those who took advantage of the holiday period to go abroad; they did not even have the possibility to go to their bank to withdraw cash. On November 20, 2009, during its first week of commercial operation on the Paris to New York route, the autopilot function of the Airbus A380, the pride of the Air France fleet, suffered a software failure where it was forced to return to New York. The passengers were dispatched to other flights. Such a software problem could have been a lot more serious. Software problems can also have an impact on an individual’s rights and freedom, be it in the United States, where voting machines failed during the presidential elections, which prevented a large number of votes from being included [THE 04]; or in France where, during local elections in 2008, a candidate from the Green party obtained 1,765 votes from 17,656 registered voters, and the software from the Ministry of Interior allowed the person to sit in the next stage of the election as the 10% threshold was reached. However, the software did not compute three digits after the decimal point and an “unfortunate rounding error to 10% was computed while the candidate only had 9.998% of the registered voters”. The end result was that the candidate was not allowed to participate in the next stage of the election. [BOU]. Software such as those listed above can be the root cause of accidents and even fatalities. This happened with the radiotherapy system Therac-25 [LEV 93, pp. 18–41], which led to six accidental releases of massive overdoses of radiation between 1985 and 1987, which led to the death of three patients. In the case of Therac-25, the root cause of the software failures – and of the death of the patients – were determined as being: a lack of code reviews by independent personnel; software design methods that were not adapted and thus incorrectly implemented for safety critical systems;  lack of awareness regarding system reliability for evaluation of software defects; ambiguous error messages and usability problems in the software; a lack of full acceptance tests for the complete system (hardware and software). There are other examples of software failures that have caused major incidents, which have occurred in the space industry, such as: the first flight of the Ariane 5 launcher, where a component that was developed and used reliably on the Ariane 4 launchers was used outside its normal operational context; this led to the loss of the launcher and all the satellites it carried; NASA’s (National Aeronautics and Space Administration) Mars Climate Orbiter mission, where a unit conversion problem, between the units used by the European Space Agency (ESA; using metric- based units) and the units used by NASA (nautical mile) led to the loss of the spaceship and the full mission; NASA’s Mars Polar Lander, where a speck of dust led to an incorrect response from one of the three landing gear, and a lack of software testing led to the shutdown of the probe’s engine some 40 meters above the surface; this led to the loss of the probe and the mission. These three examples each cost hundreds of millions of Euros and US dollars, even with the high level of quality and tests done on such systems. Every year, software failures generate financial losses evaluated to be hundreds of millions of Euros. Correct testing of software is necessary to avoid frustration, lost financial expenditure, damages to property, or even death; all this is due to failures in software. This implies effective and efficient processes throughout the development life cycle of the software and the products it operates. 1.1.2. Causes of software defects FL-1.2.3 (K2) Distinguish between root cause, error, defect and failure There is a causality link between errors and defects, and between defects and failures generated. The initial cause – the root cause – of defects is often found to be caused by the actions (or lack of action) of humans: misunderstanding of the specifications by functional analysts, resulting in a software design or architecture that prevents the expected goals from being reached or objectives stated by the customers; mistakes, such as replacing a greater than sign by a greater than or equal to sign, resulting in abnormal behavior when both variables are equal. Some failures are not directly caused by human action, but are caused by the interactions between the test object and its environment: software malfunctions when electronic components overheat abnormally due to dust; electrical or electronic interferences produced by power cables near unshielded data cables; solar storms or other activities generating ionizing radiation that impacts on electronic components (this is important for satellite and airborne equipment); impact of magnets or electromagnetic fields on data storage devices (magnetic disks or tapes, etc.). Many terms describe the incorrect behavior of software: bug, error, failure, defect, fault, mistake, etc. These terms are sometimes considered as equivalent, which may generate misunderstandings. In this book, just as for the International Software Testing Qualifications Board (ISTQB), we will use the following terms and definitions: error: human action at the root of a defect; defect: result, present in the test object, of a human action (i.e. error); failure: result from the execution of a defect by a process (whether the process is automated or not). These terminological differences are important and will result in different activities to limit their occurrence and/or impact. See also section 5.2. In order to reduce human errors – and thus the number of defects introduced in the software – training activities can be implemented, or more strict processes can be set in place. Tests are executed to identify the failures, by displaying abnormal behavior. Using the information provided by testers, designers can identify and remove defects that cause incorrect behavior. The software defects can be identified by submitting the software to reviews (code or architecture reviews, etc.) or by executing the software and identifying failures that result from the presence of defects. NOTE.– Defects may be present in software, as well as in documents and other deliverables. A large number of software problems are caused by requirements or specifications (or user stories) that may be ambiguous, or even incoherent or incompatible. The error is therefore made by those who write these requirements, the defect in the specification, and in the code, before the failure is identified during test execution. FL-1.2.1 (K2) Exemplify why testing is necessary Our software and systems become increasingly more complex, and we rely more and more on their faultless operation. Our cell phones and personal digital assistants (PDAs) are more powerful than the mainframes of 30 years ago, simultaneously integrating telephone functionalities, agenda, notepad and calendar functions, global positioning systems (GPSs), cameras, emails, instant messaging, games, voice recorders, music and video players, etc. Vehicles are equipped with progressively more electronic circuits and data processing systems (ESP (trajectory control for vehicles, anti-skid), GPS, fuel injection, airbags, course control, cruise control, etc.), and our cell phones connect automatically (via Bluetooth) to our vehicles and its audio system. We only need a small software problem and our vehicle or our cell phone becomes unusable. We also rely on other software, such as those in our credit or debit cards, where a defect can directly impact millions of users [LEM 10], such as that occurred in early 2010 when German users were victims of a major failure for over a week. We have also seen exploding virtual high-speed trains [LEP 10] (without actual victims) or the availability of customer data for rail companies available on the Internet [LET 10], as well as problems with bank software, administrations, etc. Our lives rely on software, and it is necessary to test this software. Software testing is undertaken to make sure that it works correctly, to protect against defects and potentially fatal failures. 1.1.3. Role of testing in software development, maintenance and operations Testing is – and should be – present throughout the software life cycle – whatever the life cycle is – from the beginning of its design to the end of its maintenance, and during the whole operation of the software. Rigorous testing of software and systems, including their documentation, allows a reduction in the number of defects injected in the process, reduces probability of failure during execution of the software and contributes to improving the quality of these software and systems. Tests also provide information that allows managers to make informed decisions, with a better understanding of the level of quality and the impacts of their decisions. FL-1.4.2 (K2) Explain the impact of context on the test process In the previous examples of defects (see 1.1.1 above) we can see that an identical test process would most likely not be appropriate, so we will have to ensure that the test processes, test techniques and test planning activities are appropriate to each project, and to the specific risks associated to the project. 1.1.4. Tests and quality Tests are sometimes mistaken with quality assurance and quality control. These notions are not identical: Quality assurance ensures that the organization’s processes (its best or recommended practices) are implemented and applied correctly. Continuous process improvements to increase their efficiency and their effectiveness – and thus the organizations’ efficiency and effectiveness  – and attain a higher maturity level are additional goals for quality assurance. Testing identifies defects and failures, and provides information on the software and the risks associated with their release to the market. As defects can appear anywhere in the deliverables used in software development – from inadequate requirements to incorrect implementation – it is necessary to include verification and validation of documentation as well as of software. In that regard, tests are similar to quality control. We can clearly see the complementarities of these two aspects. The software testing community is not uniform. Two main approaches are visible: In the “traditional” approach, tests are based on requirements and specifications, and the software is analyzed systematically, sometimes leading to a large volume of documentation. This approach is based on a definition and organization of the activities (test design, test case design, creation of the test data associated with the test cases, execution of the test cases on the software). This approach generates a testing workload (and a cost) before the discovery of the first defect or the first failure. In the “agile” approach, tests are executed based on the Agile Manifesto [AGI] recommendations, highlighting the search for defects associated with risks and context. This approach is based on a pragmatic evaluation of the test activities that require execution prior to software delivery. Different “flavor” exist for implementing Agile, such as Scrum, XP, SAFe, DevOps and EVO. Proponents of each approach have arguments in favor of their approach and against the other. It is a pity to see high-level professionals unable to recognize the complementarities of both approaches. One approach is more adapted to one type of project, while the other is more suited to other projects. The “traditional” or “systematic” approach is more applicable to large projects or those of a long duration, where the test team are associated relatively early with the project, even from the beginning of the design phase. This allows a more detailed analysis of the software and a test case design phase for a longer period. This approach requires sequential development cycles. The “agile” approach is more suited for qualification of smaller-sized software, and for shorter time periods. It is applicable, among others, to development cycles where the stakeholders are not 100% sure of what they require, and to rapidly evolving business environment. 1.1.5. Terminology A number of terms and concepts are used in testing, sometimes correctly sometimes incorrectly. To aid understanding, the ISTQB proposed a unique set of definitions for the terms used in software testing. Some terms, for which the definition is noted here, are noted in italic (i.e. test), and their definition, extracted from the ISTQB Glossary, is provided in the glossary of this book. In the index of this publication, the location where these terms are used is provided. The latest version of the glossary is available from the ISTQB website (www.istqb.org). In general, the definitions included in this book come from norms and international standards, or from the ISTQB Glossary of terms used in software testing. The following definitions come from the ISO 9000 [ISO 05] standard. Verification: confirmation, through the provision of objective evidence, that specified requirements have been fulfilled. “Objective evidence” is the set of data supporting the evidence or verity of something, which can be obtained through observation, measurement, testing or other means. Verification ensures that the requirements have been fulfilled, whether these requirements are applicable to the product or process. The qualifier “verified” designates the corresponding status. Therefore, verification provides a response to the question: “have we produced what is specified?” Validation: confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled. Just as for verification, “objective evidences” is a set of data supporting the evidence or reality of something, which can be obtained through observation, measurement, testing or other means. The purpose of validation is to verify that the requirements have been fulfilled for a specific intended usage. Contrary to verification, which focuses only on specified requirements, validation focuses on the usage of the validated component. Usage conditions may be real or simulated, where “validated” is the corresponding status. Therefore, “validation” provides a response to the question: “have we built the correct product?” Verification and validation are complementary but not identical. These differences will have an impact on the burden of proof to be provided by the testers. 1.2. What is testing? Testing is a set of activities with the objective of identifying failures in a software or system and to evaluate its level of quality, to obtain user satisfaction. It is a set of tasks with clearly defined goals. Detection of failures by a user, in the normal usage of the software or system, does not make a tester of that user. If a hacker studies a software to find failures and uses them to gain access to the system, that does not make that hacker a tester either. 1.2.1. Origin of defects Defects are not the product of some “gremlin” sprinkling defects in the software when the developers have their back turned away. Defects are introduced at the same time the deliverables are written, by the same persons. Figure 1.1. Origin and impacts of defects Defects and failures can arise from different root causes, such as gaps in the developer’s training, communication problems between the customer and the designers, immature design processes – from requirements gathering, to detailed design and architecture – or even oversights or misunderstanding or incorrect transcriptions of the requirements. Among other things, these causes may result from stress or fatigue of the design teams. The impact of human error on a product is called a “defect”, which will produce a “failure” (a mode of operation that does not fulfill user’s expectations) if it is executed when the software or system is used. 1.2.2. Common goals of testing FL-1.1.1 (K1) Identify typical test objectives Contrary to what some might think, testing should not be something that is done if there is time between the end of the design activities and the delivery of the product. Over the years, testing has seen its goals change and evolve. From verification that the software works correctly, the software underwent verification that it did not have defects, through to a phase where it delivers information that allows decisions to be made; now, it has become a set of techniques and methods that enables decisions for delivery to users, taking into account the objectives of cost reduction, time to market and risks. Software testing focuses on two complementary but distinct aspects: defect and failure detection, so that these can be fixed and thus the quality of the product delivered to customers and users is improved; decisions to be made on the basis of the information provided regarding the level of risks associated with the delivery of the software to the market, and on the efficiency of the organization’s processes which are the root cause of the identified defects and/or failures. 1.2.3. Examples of objectives for testing FL-1.2.2 (K1) Recall the relation between testing and quality Test objectives vary, depending on the phase of the life cycle of the software. The objectives are not identical during the initial design, maintenance, or at the end of the software usage. Similarly, they differ also according to the test level. During the general design and detailed design phases, testing focuses on finding the highest number of defects (or failures), in the shortest possible timescale, in order to deliver high-quality software. During the customer acceptance phase, testing will show that the software works properly to obtain customer approval for usage of that software. During the operational phases, where the software is being used, testing will focus on ensuring that the requirement levels (SLA: service-level agreement, explicit or implicit) are reached. During evolutive or corrective maintenance of the software, testing aims to ensure the absence of defects via corrections or evolutions, and also that no side effects (regression) occur on the unchanged functionalities of the system or software. When the software is discarded and replaced by another, testing takes a snapshot of the software, to ensure which functionalities are present and guarantee the quality of data, so that migration goes smoothly to another platform, whether this is new hardware or new software. Data transfer from the old to the new environment is also important and must be tested. Therefore, we see that the testing objectives vary, depending on the phase of the software’s life cycle. 1.2.4. Test and debugging FL-1.1.2 (K2) Differentiate testing from debugging Testing is the identification of one or more characteristics according to a defined procedure [ISO 05]. The characteristics can be inherent or implied, qualitative or quantitative, and grouped according to different criteria (physical, sensory, behavioral, temporal, ergonomic, functional, etc.). A procedure is defined here as a specified way to carry out an activity or process. The result from test execution is that the characteristic is present or is not. There are many aspects associated with what is usually called a test: an objective (the characteristic that we want to ascertain); a way to determine the characteristic (the defined procedure and its execution); an activity (the execution of the procedure to obtain a result); a result (the presence or absence of the characteristic to the expected level). More precise definitions and terms will be provided later in this book. Generally, when we determine that an expected characteristic is not present, we will talk of a “failure”. Failures are the result of the execution of a defective piece of code, of a defect in the software. To remove the failure, it is necessary to fix the defect. Developers fix software defects, which is called “debugging” and consists of finding and removing the exact cause of the failure. This is not a task involved in the testing process. 1.3. Paradoxes and main principles FL-1.3.1 (K2) Explain the seven principles in testing During the last 50 years, a number of major principles have been identified, that apply to any test project, regardless of their environment. 1.3.1. Testing identifies the presence of defects Testing enables the identification of defects present in a piece of code, but does not show that such defects are not present. In order to demonstrate absence of defects, all combinations of all possible actions on all reachable objects of the software, for all combinations of input data, in all possible contexts have to be tested, i.e. hardware: mother board, central processing unit (CPU) and number of CPUs, bus, network, random access memory (RAM), I/O speed, hard disk drive speed and capacity, and software (operating system and its parameter settings, other software that could interact with the software, either at a software or hardware level, etc.); this is, to all intent and purpose, impossible. It is impossible to demonstrate the absence of defect in software, but only the presence of such defects. Test execution of software enables the reduction of risks of residual – unidentified – defects, but cannot guarantee that all defects have been identified. 1.3.2. Exhaustive testing is impossible Second principle: it is impossible to undertake exhaustive testing, to test “everything”. When testing a small calculator, “testing everything” would mean trying all combinations of input values, actions, configurations (hardware and software), resulting in an almost infinite number of test cases, which is not realistic to design or execute. Exhaustive testing is not possible, except in some rare trivial cases. We must reduce the number of tests that are designed and/or executed, so as to obtain a number of tasks that it is economically possible to execute. Therefore, this involves choosing which tests to design and execute. This is risk management. The test team activities are software risk limitation (mitigation) activities. 1.3.3. Early testing Third principle: test early in the development cycle, or “early testing”. This principle is based on the fact that costs, whether development costs, testing costs, defect correction costs, etc. increase throughout the project. It is economically more sensible to detect defects as early as possible, thus avoiding the design of incorrect code, the creation of test cases, which require design and maintenance of the created test cases, the identification of defects that have to be logged, fixed and retested. A ratio usually applied in industry (validated numerous times in Europe, USA and Asia) is as follows: for a specific unit of cost, the cost of finding (and fixing) a defect in the design phase is “1”. If the defect is identified in the coding phase, it is multiplied by “10”. If the defect is found during the test phase (system test or acceptance test), then the cost is multiplied by “100”. Finally, if the defect is found in production (by the customer or by a user), then the cost is multiplied by “1,000”. Therefore, it is economically more efficient to find defects as early as possible. NOTE.– Calculation of the return on investment of testing is either empirical or based on statistics of the cost of fixing defects (assuming you have measured this cost over time in your organization). Empirically, it is clear that fixing a requirement (i.e. changing a minus sign to a plus sign in a requirement specification) is easier and cheaper during the requirements review than after the code, or the tests have been designed and created. For defects found in the field, it is necessary to take into account the cost of restoring the software and data on the customer site (including installation, cost of travel for the technician, etc.), and also any loss of business or of prospective customers, because of bad publicity, etc. 1.3.4. Defect clustering Fourth principle: defect aggregation. Even if bugs do not have a real life, it is frequent that numerous defects are concentrated according to some identical criteria. These can be a piece of code (such as a complex section of an algorithm), a sub-assembly of components designed by the same person (where a similar error is repeated) or a group of components designed in the same period of time; it can even a defective off-the-shelf component (COTS). This principle means also that if you find a defect, it might be efficient to look for other defects in the same piece of code. 1.3.5. Pesticide paradox Fifth principle: the pesticide paradox, or lack of efficiency of tests when they are used over a period of time. This loss of efficiency is due to the re-execution of identical tests (same data, same actions) on software paths that have already been tested. If a failure has been identified and corrected during a previous execution of that test, then it will not be detected anymore during the re-execution of that test at a later time. Other defects could be evidenced, aside from those already identified. In the end, the re-execution of that test will work without problem and will not identify any additional defect. Test efficiency will decrease, just as pesticide efficiency decreases over time and usage. It is strongly suggested that tests are changed over time, so as to modify them and so that the impact of this principle will be negligible. Uses of other test techniques, variation in test data, or in the order of execution of the tests are ways to counter this principle. Reuse, without modification, of tests can be necessary to ensure that no side effects (i.e. regression) have been introduced in the software as a result of changes in that software or in its execution context. Such activities are called regression tests. 1.3.6. Testing is context dependent Sixth principle: testing is dependent on the context. Safety-critical systems will be tested differently and with a more intense effort, than most e-commerce systems. However, this principle goes much further, because it also applies to the evolution of the context of your own software application. During the first tests of your software, you will design tests based on your knowledge of the software at that moment in time. During subsequent tests, or tests of the next version of the software, your knowledge of the software evolves, and you will design other tests, focusing, for example, on identified risks, on components where defect clustering occurred, or taking into account the pesticide paradox. Your test effort will be influenced by the available time and resources, by previous defects, code quality and the objectives you have been assigned. Reuse of test plans, test design, test conditions or test cases from other software or projects is therefore counterproductive, even though it might seem intuitively valid. This does not mean that you should not take heed of the experience acquired by others, but that you should – at all times – be aware of your own environment, of its evolution, and any mandatory adaptation to your context. 1.3.7. Absence-of-errors fallacy Seventh principle: absence-of-errors fallacy. Frequently, the reduction of number of defects is considered to be the ultimate goal of testing. However, it is not possible to assure that a software application, even if it has no defect, suits the needs of the public. If the software does not correspond to what the users need or to their expectations, identification and correction of defects will not help the architecture or the usability of the software, nor the financial success of the software. 1.4. Test activities, testware and test roles Testing is context dependent, but, at a high level, there are common sets of test activities without which testing will not achieve the test objectives. These sets of test activities form a test process. The test process should be tailored to a given situation based on various factors specific to that situation. Which test activities are included in this test process, how they are implemented, and when they occur should normally be decided as part of the test planning for the specific situation. We will see the general aspects of the test process in terms of test activities and tasks, the impact of context, testware, traceability between the test basis and testware, and testing roles. The ISO/IEC/IEEE 29119-2 standard provides further information about test processes. FL-1.4.1 (K2) Summarize the different test activities and tasks Testing is not limited to the execution of software with the aim of finding failures: it is also necessary to plan, define goals, identify test conditions, create test data, start and exit criteria, test environments and, of course, control all these activities. Figure 1.2. Fundamental test processes These activities are grouped in a number of major fundamental processes: test planning; test monitoring and control of the test activities; analysis of work products and creation of test conditions; design of tests conditions and of test cases; implementation of tests cases – manual or automated – in the test environment;  execution of tests, evaluation of exit criteria and production of test reports; test completion activities. Test infrastructure management is not included in the ISTQB syllabus, but is something to take into account as you should not do tests in an uncontrolled – or a production – environment. These activities are repeated at each test level and for each test campaign (each iteration), regardless of the software or system to be tested. 1.4.1. Planning Before any activity, it is beneficial to organize and plan the activities to be executed. This also applies for software and system testing. Test planning is detailed in Chapter 5, and consists of defining test objectives, identifying constraints and defining the activities to reach these goals. Test planning activities include organization of the tasks, and coordination with the other stakeholders, such as development teams, support teams, user representatives, management and customers. The level of detail will depend on the context: a complex safety-critical software will not be tested with the same goals or the same focus as a video game or e-commerce software. 1.4.1.1. Planning work products The following are the usual work products available at the end of test planning: one or more test plans; risks and dependencies of the tasks identified in the test plans. 1.4.2. Monitoring and control Test monitoring focuses on current test activities (such as measuring their progress) as well as future test activities (ensuring these can start on time), and comparing actual progress to the planned progress identified in the planning activity. Test monitoring and test control activities are executed throughout the test campaigns and are often grouped with project planning activities to ensure that what has been planned is correctly implemented. Control identifies deviations from planned operations, or variations with regards to planned objectives, and proposes actions to reach these objectives. This implies the ability to measure the progress of activities in terms of the resources used (including time), as well as in terms of objectives reached. Similar to test planning activities, test control activities are described in Chapter 5 in more detail. 1.4.2.1. Monitoring and control work products The following are the usual work products produced during monitoring and control: test reports, produced periodically summarizing the test progress; statistics on defects (detection and fix trends, defects per module, test execution coverage per module, defects age, number of reopened defects, defect detection percentage); risks and dependencies of the tasks identified with relation to the test plans. 1.4.3. Test analysis and design The analysis and design phase is where global test objectives, as defined by the organization during the test planning activity, are used to create the test conditions for the software. The high-level test objectives, identified during the planning phase, are used to create the test design documents, test conditions and test procedures. These activities are described in Chapter 5 and include analysis of the test basis as well as the creation of high-level test conditions. 1.4.3.1. Analysis of the test basis Analysis of the test basis is the study of the reference documents used to design the software and the test objectives in order to ensure that nothing is forgotten. This includes, but is not limited to: contractual documents, such as the contract, statement of work and any amendments or technical attachments; software specifications, high-level design, detailed architecture of the components, database organization or file system organization; user documentation, maintenance or installation manuals, etc.; the risks identified or associated with the use or the development of the software; applicable standards, whether they be company-specific, profession- specific or mentioned in the contractual documents. A test basis that requires a specific or formal process, such as negotiations, contract change notice or other similar documentation, before it can be changed is called a frozen test basis. The analysis of the test basis allows the definition of the objectives of the test, their prioritization and the evaluation of the testability of the test basis and test objectives. During this phase, we will identify the risks and test priorities (integrity level), as well as test environments (including test data) to be acquired. We will also select the measurements and metrics that will allow us to measure test progress. The integrity level indicates the criticality of the software for the stakeholders, and is based on attributes of the software, such as risks, safety and security level. The integrity level impacts the depth and breadth of tests to be executed, type and the level of detail of test documentation and the minimal set of testing tasks to be executed. Integrity levels, as defined by the IEEE 829-2008 [IEE 08a, pp. 13–14] standards, are: Level 4: catastrophic: software must execute correctly or grave consequences (loss of life, loss of system, environmental damage, economic or social loss) will occur. No mitigation is possible.  Level 3: critical: software must execute correctly or the intended use (mission) of system/software will not be realized, causing serious consequences (permanent injury, major system degradation, environmental damage, economic or social impact). Partial-to- complete mitigation is possible. Level 2: marginal: software must execute correctly or the intended function will not be realized causing minor consequences. Complete mitigation possible. Level 1: negligible: software must execute correctly or the intended function will not be realized causing negligible consequences. Mitigation not required. IEEE 829-2008 integrity levels can be linked to the failure conditions categorizations described in DO178B/ED12B [EUR 00, pp. 7–8]: catastrophic (category A): failure conditions that would prevent continued safe flight and landing; hazardous/severe-major (category B): failure conditions that would reduce the capability of the aircraft or the ability of the crew to cope with adverse operating conditions to the extent that there would be: - a large reduction in safety margins or functional capabilities, - physical distress or higher workload, such that the flight crew could not be relied on to perform their tasks accurately or completely, - adverse effects on occupants, including serious or potentially fatal injuries to a small number of those occupants; major (category C): failure conditions that would reduce the capability of the aircraft or the ability of the crew to cope with adverse operating conditions to the extent that there would be; for example, a significant reduction in safety margins or functional capabilities; a significant increase in crew workload or in conditions impairing crew efficiency or discomfort to occupants, possibly including injuries; minor (category D): failure conditions that would not significantly reduce aircraft safety, and that would involve crew actions that are well  within their capabilities. Minor failure conditions may include, for example, a slight reduction in safety margins or functional capabilities, a slight increase in crew workload, such as routine flight plan changes, or some inconvenience to occupants; without effect (category E): failure conditions that do not affect the operational capability of the aircraft or increase crew workload. Analysis of the test basis allows you to identify the aspects to test (test objectives) and to determine how they will be tested. Traceability from the test basis to the test objectives and test conditions, to allow quick impact analysis of change requests to the test basis, should be ensured. 1.4.3.2. Test design Test objectives correspond to the reasons (or goals) that we have, during test design and execution. These guide our actions and lead to the detailed design of the test cases. Test design consists of applying the test objectives under obvious test conditions and then applying them to test cases. Test conditions are usually abstract, while test cases are usually precise and include both test data and expected results. For more details on the difference between the design of test conditions and test cases, see Chapter 4. The test design phase comprises: identification and prioritization of the test conditions based on an analysis of the test objects, of the structure of the software and system; design and prioritization of high-level test cases; identification of the test environment and test data required for test execution; provision for control and tracking information that will enable evaluation of test progress. Bidirectional traceability between the test basis, the requirements and risks on one hand, and the test conditions on the other hand, enables all the requirements and all the risks to be covered by the tests; and that all test conditions are attached to a higher-level test objective. It is also important to ensure that traceability is maintained up to the test case execution and the defects. This will facilitate monitoring and control throughout the project, and help in communicating with stakeholders. The integrity level enables the approximation of the depth and breadth of the test to execute, and thus helps to devise the test conditions. 1.4.3.3. Test analysis and test design work products The following are the work products produced during test analysis: defined and prioritized test conditions; bidirectional traceability between the elements of the test basis and the test conditions; test charters for exploratory testing; defects identified in the test basis and suggestions for the test basis and test items; risks and dependencies identified with relation to the test conditions. The following are the work products produced during test design: defined and prioritized test cases that exercise the test conditions; bidirectional traceability between the test cases and the test conditions; test environment design, including definition of specific test data; list of required infrastructure and tools; refined test conditions (where necessary). 1.4.4. Test implementation Test implementation is the conversion of higher-level test conditions to lower-level test cases and test procedures, with specific test data and precise expected results. Detailed information on test environment and test data, as well as on the sequencing of the test cases, are necessary to anticipate test execution. Test implementation tasks include (non-exhaustive list):  finalizing, implementing and ordering test cases based on the priorities defined. This can come from the integrity levels or other considerations, such as risk analysis or the relative criticality of the components; developing and sequencing the test procedures, by organizing test cases and test data. This can require the creation of drivers or stubs, or even automated test cases; creating test suites (scenarios) from test procedures and test cases to facilitate test execution; defining the test environment and designing test data; ensuring that bidirectional traceability, started during the analysis and test design phases, is continued until the test case levels; providing information on the evolution of the process (metrics and measurement) so that project control and management can be efficient. A test case is defined by a starting environment, input data, actions and expected results, which include the expected data and resulting environment. Last minute changes, such as to the test environment or a reorganization of priorities will modify the test cases, test procedures or test suites, or even the test data. Full and detailed traceability will save time – via an impact analysis – and ensure tracking and control of the tests in such difficult conditions. 1.4.4.1. Test implementation work products The following are the work products produced during test implementation: test case and test procedure sequencing; test suites; test execution schedule; demonstrable achievement of coverage criteria via bidirectional traceability; test environment(s);  test data; concrete test case values (expected results); refined test conditions (possible). 1.4.5. Test execution Test execution in the test environment enables the identification of differences between the expected and the actual results, and includes tasks linked to the execution of test cases, test procedures or test suites. This includes: ensuring that the test environment is ready for use, including the availability of test data; executing test cases, test procedures and test suites, either manually or automatically, according to the planned sequence and priority; recording test execution results and identifying the version of the component tested, of the test tools, and test environment; comparing expected results with actual results; identifying and analyzing any discrepancy between expected and actual results, and clearly defining the cause of the discrepancy. Recording these differences as incidents or defects, by providing the highest level of detail possible to facilitate defect correction in the future; providing tracking and control information to allow efficient management of test activities. Test execution activities include activities re-executed for each fix of identified defects. It is necessary to ensure that the fixes have corrected the defect (confirmation testing or retest) and that the modifications introduced have not generated side effects or regressions (regression test). Traceability initiated in earlier phases must continue, so that these can be associated with the test objectives, test conditions and test cases, information on test execution and obtained test results. In the case of the presence of defects, this will allow determination of the integrity level associated with the test condition, and thus the importance and impact of the defect as seen by the user, and the need for urgent correction of this defect. We must also determine when to stop with test execution. This is done by analyzing the exit criteria to evaluate if the test objectives and criteria defined during the planning phase have been complied with. This includes: analysis of the test execution logs and reports (notes taken during test execution); comparison of the objectives reached versus objectives identified during the planning phase, and evaluation of the need to test more thoroughly or to modify the exit criteria. If one or more of the exit criteria have not been reached after execution of the planned tests, additional tests must be designed and executed to ensure that these new tests enable the intended exit criteria to be reached. A detailed analysis can identify areas where there is no justification to undertake further tests in order to reach the expected exit criteria, such as exceptional or unreachable situations, or in the case of dead code. 1.4.5.1. Test execution work products The following are the work products produced during test execu