Software Testing—Component Level PDF

CHAPTER 19 Software Testing—Component Level Software component testing incorporates a strategy that describes the steps to be conducted as part of testing, when these steps are planned and then undertaken,...

CHAPTER 19 Software Testing—Component Level Software component testing incorporates a strategy that describes the steps to be conducted as part of testing, when these steps are planned and then undertaken, and how much effort, time, and resources will be required. Within the testing strategy, software component testing implements a collection of component test- ing tactics that address test planning, test-case design, test execution, and resul- tant data collection and evaluation. Both component testing strategy and tactics are considered in this chapter. Key basis path testing...................... 384 interface testing....................... 388 Concepts black-box testing...................... 388 object-oriented testing................. 390 boundary value analysis................ 389 scaffolding........................... 379 class testing.......................... 390 system testing........................ 376 control structure testing................ 386 testing methods....................... 373 cyclomatic complexity.................. 385 testing strategies...................... 373 debugging........................... 373 unit testing........................... 376 equivalence partitioning................ 389 validation testing...................... 376 independent test group................. 375 verification........................... 373 integration testing..................... 375 white-box testing...................... 383 Quick Look What is it? Software is tested to uncover errors single component or on a small group of re- that were made inadvertently as it was de- lated components and applies tests to signed and constructed. A software compo- uncover errors in the data and processing nent testing strategy considers testing of logic that have been encapsulated by the individual components and integrating them component(s). After components are tested, into a working system. they must be integrated until the complete Who does it? A software component testing system is constructed. strategy is developed by the project manager, What is the work product? A test specifica- software engineers, and testing specialists. tion documents the software team’s approach Why is it important? Testing often accounts to testing by defining a plan that describes an for more project effort than any other software overall strategy and a procedure that defines engineering action. If it is conducted haphaz- specific testing steps and the types of test ardly, time is wasted, unnecessary effort is ex- cases that will be conducted. pended, and even worse, errors sneak How do I ensure that I’ve done it right? An through undetected. effective test plan and procedure will lead to What are the steps? Testing begins “in the the orderly construction of the software and small” and progresses “to the large.” By this the discovery of errors at each stage in the we mean that early testing focuses on a construction process. 372  CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 373 To be effective, a component testing strategy should be flexible enough to promote a customized testing approach but rigid enough to encourage reasonable planning and management tracking as the project progresses. Component testing remains a respon- sibility of individual software engineers. Who does the testing, how engineers com- municate their results with one another, and when testing is done is determined by the software integration approach and design philosophy adopted by the development team. These “approaches and philosophies” are what we call strategy and tactics—topics to be discussed in this chapter. In Chapter 20 we discuss the integration testing techniques that often end up defining the team development strategy. 19.1 A S t r at e g i c A p p r oac h to S o f t wa r e T e s t i n g Testing is a set of activities that can be planned in advance and conducted system- atically. For this reason, a template for software testing—a set of steps into which we can place specific test-case design techniques and testing methods—should be defined for the software process. A number of software testing strategies have been proposed in the literature [Jan16] [Dak14] [Gut15]. All provide you with a template for testing, and all have the following generic characteristics: ∙ To perform effective testing, you should conduct technical reviews (Chapter 16). By doing this, many errors will be eliminated before testing commences. ∙ Testing begins at the component level and works “outward” toward the integration of the entire computer-based system. ∙ Different testing techniques are appropriate for different software engineering approaches and at different points in time. ∙ Testing is conducted by the developer of the software and (for large projects) an independent test group. ∙ Testing and debugging are different activities, but debugging must be accom- modated in any testing strategy. A strategy for software testing incorporates a set of tactics that accommodate the low-level tests necessary to verify that a small source code segment has been correctly implemented as well as high-level tests that validate major system functions against customer requirements. A strategy should provide guidance for the practitioner and a set of milestones for the manager. Because the steps of the test strategy occur at a time when deadline pressure begins to rise, progress must be measurable and problems should surface as early as possible. 19.1.1 Verification and Validation Software testing is one element of a broader topic that is often referred to as verifica- tion and validation (V&V). Verification refers to the set of tasks that ensure that software correctly implements a specific function. Validation refers to a different set 374 PART THREE QUALIT Y AN D S ECURIT Y of tasks that ensure that the software that has been built is traceable to customer requirements. Boehm [Boe81] states this another way: Verification: “Are we building the product right?” Validation: “Are we building the right product?” The definition of V&V encompasses many software quality assurance activities (Chapter 19).1 Verification and validation include a wide array of SQA activities: technical reviews, quality and configuration audits, performance monitoring, simulation, feasi- bility study, documentation review, database review, algorithm analysis, development testing, usability testing, qualification testing, acceptance testing, and installation test- ing. Although testing plays an extremely important role in V&V, many other activities are also necessary. Testing does provide the last bastion from which quality can be assessed and, more pragmatically, errors can be uncovered. But testing should not be viewed as a safety net. As they say, “You can’t test in quality. If it’s not there before you begin testing, it won’t be there when you’re finished testing.” Quality is incorporated into software throughout the process of software engineering, and testing cannot be applied as a fix at the end of the process. Proper application of methods and tools, effective technical reviews, and solid management and measurement all lead to quality that is confirmed during testing. 19.1.2 Organizing for Software Testing For every software project, there is an inherent conflict of interest that occurs as test- ing begins. The people who have built the software are now asked to test the software. This seems harmless in itself; after all, who knows the program better than its devel- opers? Unfortunately, these same developers have a vested interest in demonstrating that the program is error-free, that it works according to customer requirements, and that it will be completed on schedule and within budget. Each of these interests mitigates against thorough testing. From a psychological point of view, software analysis and design (along with cod- ing) are constructive tasks. The software engineer analyzes, models, and then creates a computer program and its documentation. Like any builder, the software engineer is proud of the edifice that has been built and looks askance at anyone who attempts to tear it down. When testing commences, there is a subtle, yet definite, attempt to “break” the thing that the software engineer has built. From the point of view of the builder, testing can be considered to be (psychologically) destructive. So the builder treads lightly, designing and executing tests that will demonstrate that the program works, rather than to uncover errors. Unfortunately, errors will be nevertheless present. And, if the software engineer doesn’t find them, the customer will! 1 It should be noted that there is a strong divergence of opinion about what types of testing constitute “validation.” Some people believe that all testing is verification and that validation is conducted when requirements are reviewed and approved, and later, by the user when the system is operational. Other people view unit and integration testing (Chapters 19 and 20) as verification and higher-order testing (Chapter 21) as validation.  CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 375 There are often a number of misconceptions that you might infer from the preced- ing discussion: (1) that the developer of software should do no testing at all, (2) that the software should be “tossed over the wall” to strangers who will test it mercilessly, and (3) that testers get involved with the project only when the testing steps are about to begin. Each of these statements is incorrect. The software developer is always responsible for testing the individual units (com- ponents) of the program, ensuring that each performs the function or exhibits the behavior for which it was designed. In many cases, the developer also conducts inte- gration testing—a testing step that leads to the construction (and test) of the complete software architecture. Only after the software architecture is complete does an inde- pendent test group become involved. The role of an independent test group (ITG) is to remove the inherent problems associated with letting the builder test the thing that has been built. Independent test- ing removes the conflict of interest that may otherwise be present. After all, ITG personnel are paid to find errors. However, you don’t turn the program over to ITG and walk away. The developer and the ITG work closely throughout a software project to ensure that thorough tests will be conducted. While testing is conducted, the developer must be available to correct errors that are uncovered. The ITG is part of the software development project team in the sense that it becomes involved during analysis and design and stays involved (planning and specify- ing test procedures) throughout a large project. However, in many cases the ITG reports to the software quality assurance organization, thereby achieving a degree of indepen- dence that might not be possible if it were a part of the software engineering team. 19.1.3 The Big Picture The software process may be viewed as the spiral illustrated in Figure 19.1. Ini- tially, system engineering defines the role of software and leads to software require- ments analysis, where the information domain, function, behavior, performance, Figure 19.1 Testing System testing strategy Validation testing Integration testing Unit testing Code Design Requirements System engineering 376 PART THREE QUALIT Y AN D S ECURIT Y constraints, and validation criteria for software are established. Moving inward along the spiral, you come to design and finally to coding. To develop computer software, you spiral inward along streamlines that decrease the level of abstraction on each turn. A strategy for software testing may also be viewed in the context of the spiral (Figure 19.1). Unit testing begins at the vortex of the spiral and concentrates on each unit (e.g., component, class, or WebApp content object) of the software as imple- mented in source code. Testing progresses by moving outward along the spiral to integration testing, where the focus is on design and the construction of the software architecture. Taking another turn outward on the spiral, you encounter validation testing, where requirements established as part of requirements modeling are vali- dated against the software that has been constructed. Finally, you arrive at system testing, where the software and other system elements are tested as a whole. To test computer software, you spiral out along streamlines that broaden the scope of testing with each turn. Considering the process from a procedural point of view, testing within the context of software engineering is actually a series of four steps that are implemented sequen- tially. The steps are shown in Figure 19.2. Initially, tests focus on each component individually, ensuring that it functions properly as a unit. Hence, the name unit testing. Unit testing makes heavy use of testing techniques that exercise specific paths in a component’s control structure to ensure complete coverage and maximum error detec- tion. Next, components must be assembled or integrated to form the complete software package. Integration testing addresses the issues associated with the dual problems of verification and program construction. Test-case design techniques that focus on inputs and outputs are more prevalent during integration, although techniques that exercise specific program paths may be used to ensure coverage of major control paths. Figure 19.2 Software testing steps Requirements High-order tests Design Integration test Code Unit test Testing "direction"  CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 377 After the software has been integrated (constructed), a set of high-order tests is con- ducted. Validation criteria (established during requirements analysis) must be evalu- ated. Validation testing provides final assurance that software meets all functional, behavioral, and performance requirements. The last high-order testing step falls outside the boundary of software engineering and into the broader context of computer system engineering (discussed in Chapter 21). Software, once validated, must be combined with other system elements (e.g., hard- ware, people, databases). System testing verifies that all elements mesh properly and that overall system function and performance is achieved. S afe H ome Preparing for Testing The scene: Doug Miller’s office, Ed: It is. Even though we’re not using Extreme as component-level design con- Programming per se, we decided that it’d be a tinues and construction of certain components good idea to design unit tests before we build begins. the component—the design gives us all of the The players: Doug Miller, software engineer- information we need. ing manager; Vinod, Jamie, Ed, and Shakira, Jamie: I’ve been doing the same thing. members of the SafeHome software Vinod: And I’ve taken on the role of the inte- engineering team. grator, so every time one of the guys passes a The conversation: component to me, I’ll integrate it and run a Doug: It seems to me that we haven’t spent series of regression tests [see Section 20.3 for enough time talking about testing. a discussion on regression testing] on the partially integrated program. I’ve been working Vinod: True, but we’ve all been just a little to design a set of appropriate tests for each busy. And besides, we have been thinking function in the system. about it... in fact, more than thinking. Doug (to Vinod): How often will you run the Doug (smiling): I know... we’re all overloaded, tests? but we’ve still got to think down the line. Vinod: Every day... until the system is Shakira: I like the idea of designing unit tests integrated... well, I mean until the software before I begin coding any of my components, increment we plan to deliver is integrated. so that’s what I’ve been trying to do. I have a pretty big file of tests to run once code for my Doug: You guys are way ahead of me! components is complete. Vinod (laughing): Anticipation is everything in Doug: That’s an Extreme Programming [an the software biz, Boss. agile software development process, see Chapter 3] concept, no? 19.1.4 Criteria for “Done” A classic question arises every time software testing is discussed: “When are we done testing—how do we know that we’ve tested enough?” Sadly, there is no definitive answer to this question, but there are a few pragmatic responses and early attempts at empirical guidance. 378 PART THREE QUALIT Y AN D S ECURIT Y One response to the question is: “You’re never done testing; the burden simply shifts from you (the software engineer) to the end user.” Every time the user executes a computer program, the program is being tested. This sobering fact underlines the importance of other software quality assurance activities. Another response (somewhat cynical but nonetheless accurate) is: “You’re done testing when you run out of time or you run out of money.” Although few practitioners would argue with these responses, you need more rigor- ous criteria for determining when sufficient testing has been conducted. The statistical quality assurance approach (Section 17.6) suggests statistical use techniques [Rya11] that execute a series of tests derived from a statistical sample of all possible program executions by all users from a targeted population. By collecting metrics during soft- ware testing and making use of existing statistical models, it is possible to develop meaningful guidelines for answering the question: “When are we done testing?” 19.2 P la n n i n g and Recordkeeping Many strategies can be used to test software. At one extreme, you can wait until the system is fully constructed and then conduct tests on the overall system in the hope of finding errors. This approach, although appealing, simply does not work. It will result in buggy software that disappoints all stakeholders. At the other extreme, you could conduct tests on a daily basis, whenever any part of the system is constructed. A testing strategy that is chosen by many software teams (and the one we recom- mend) falls between the two extremes. It takes an incremental view of testing, begin- ning with the testing of individual program units, moving to tests designed to facilitate the integration of the units (sometimes on a daily basis), and culminating with tests that exercise the constructed system as it evolves. The remainder of this chapter will focus on component-level testing and test-case design. Unit testing focuses verification effort on the smallest unit of software design—the software component or module. Using the component-level design description as a guide, important control paths are tested to uncover errors within the boundary of the module. The relative complexity of tests and the errors those tests uncover is limited by the constrained scope established for unit testing. The unit test focuses on the internal processing logic and data structures within the boundaries of a component. This type of testing can be conducted in parallel for multiple components. The best strategy will fail if a series of overriding issues are not addressed. Tom Gilb [Gil95] argues that a software testing strategy will succeed only when software testers: (1) specify product requirements in a quantifiable manner long before testing commences, (2) state testing objectives explicitly, (3) understand the users of the software and develop a profile for each user category, (4) develop a testing plan that emphasizes “rapid cycle testing,”2 (5) build “robust” software that is designed to test 2 Gilb [Gil95] recommends that a software team “learn to test in rapid cycles (2 percent of project effort) of customer-useful, at least field ‘trialable,’ increments of functionality and/ or quality improvement.” The feedback generated from these rapid cycle tests can be used to control quality levels and the corresponding test strategies. CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 379 itself (the concept of antibugging is discussed briefly in Section 9.3), (6) use effective technical reviews as a filter prior to testing, (7) conduct technical reviews to assess the test strategy and test cases themselves, and (8) develop a continuous improvement approach (Chapter 28) for the testing process. These principles are reflected in agile software testing as well. In agile develop- ment, the test plan needs to be established before the first sprint meeting and reviewed by stakeholders. This plan merely lays out the rough time line, standards, and tools to be used. The test cases and directions for their use are developed and reviewed by the stakeholders as the code needed to implement each user story is created. Testing results are shared with all team members as soon as practical to allow changes in both existing and future code development. For this reason, many teams choose to keep their test recordkeeping in online documents. Test recordkeeping does not need to be burdensome. The test cases can be recorded in a Google Docs spreadsheet that briefly describes the test case, contains a pointer to the requirement being tested, contains expected output from the test case data or the criteria for success, allows testers to indicate whether the test was passed or failed and the dates the test case was run, and should have room for comments about why a test may have failed to aid in debugging. This type of online form can be viewed as needed for analysis, and it is easy to summarize at team meetings. Test-case design issues are discussed in Section 19.3. 19.2.1 Role of Scaffolding Component testing is normally considered as an adjunct to the coding step. The design of unit tests can occur before coding begins or after source code has been generated. A review of design information provides guidance for establishing test cases that are likely to uncover errors. Each test case should be coupled with a set of expected results. Because a component is not a stand-alone program, some type of scaffolding is required to create a testing framework. As part of this framework, driver and/or stub software must often be developed for each unit test. The unit-test environment is illustrated in Figure 19.3. In most applications a driver is nothing more than a “main program” that accepts test-case data, passes such data to the component (to be tested), and prints relevant results. Stubs serve to replace modules that are subordinate (invoked by) the component to be tested. A stub or “dummy subprogram” uses the subordinate module’s interface, may do minimal data manipulation, prints verification of entry, and returns control to the module undergoing testing. Drivers and stubs represent testing “overhead.” That is, both are software that must be coded (formal design is not commonly applied) but that is not delivered with the final software product. If drivers and stubs are kept simple, actual overhead is rela- tively low. Unfortunately, many components cannot be adequately unit-tested with “simple” scaffolding software. In such cases, complete testing can be postponed until the integration test step (where drivers or stubs are also used). 19.2.2 Cost-Effective Testing Exhaustive testing requires every possible combination of input values and test-case orderings be processed by the component being tested (e.g., consider the move gen- erator in a computer chess game). In some cases, this would require the creation of 380 PART THREE QUALIT Y AN D S ECURIT Y Figure 19.3 Unit-test environment Interface Local data structures Boundary conditions Independent paths Error-handling paths RESULTS a near-infinite number of data sets. The return on exhaustive testing is often not worth the effort, since testing alone cannot be used to prove a component is correctly imple- mented. There are some situations in which you will not have the resources to do comprehensive unit testing. In these cases, testers should select modules crucial to the success of the project and those that are suspected to be error-prone because they have complexity metrics as the focus for your unit testing. Some techniques for minimizing the number of test cases required to do a good job testing are discussed in Sections 19.4 through 19.6. I nfo Exhaustive Testing such processor exists) has been developed for Consider a 100-line program in the lan- exhaustive testing. The processor can develop guage C. After some basic data declara- a test case, execute it, and evaluate the results tion, the program contains two nested loops that in one millisecond. Working 24 hours a day, execute from 1 to 20 times each, depending on 365 days a year, the processor would work for conditions specified at input. Inside the interior 3170 years to test the program. This would, loop, four if-then-else constructs are required. undeniably, cause havoc in most development There are approximately 1014 possible paths that schedules. may be executed in this program! Therefore, it is reasonable to assert that ex- To put this number in perspective, we assume haustive testing is impossible for large software that a magic test processor (“magic” because no systems.  CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 381 19.3 T e s t -C a s e D e s i g n It is a good idea to design unit test cases before you develop code for a component. This ensures that you’ll develop code that will pass the tests or at least the tests you thought of already. Unit tests are illustrated schematically in Figure 19.4. The module interface is tested to ensure that information properly flows into and out of the program unit under test (Section 19.5.1). Local data structures are examined to ensure that data stored tempo- rarily maintains its integrity during all steps in an algorithm’s execution. All indepen- dent paths through the control structure are exercised to ensure that all statements in a module have been executed at least once (Section 19.4.2). Boundary conditions are tested to ensure that the module operates properly at boundaries established to limit or restrict processing (Section 19.5.3). And finally, all error-handling paths are tested. Data flow across a component interface is tested before any other testing is initi- ated. If data do not enter and exit properly, all other tests are moot. In addition, local data structures should be exercised and the local impact on global data should be ascertained (if possible) during unit testing. Selective testing of execution paths is an essential task during the unit test. Test cases should be designed to uncover errors due to erroneous computations, incorrect comparisons, or improper control flow. Boundary testing is one of the most important unit-testing tasks. Software often fails at its boundaries. That is, errors often occur when the nth element of an n-dimensional array is processed, when the ith repetition of a loop with i passes is invoked, or when the maximum or minimum allowable value is encountered. Test cases that exercise Figure 19.4 Unit test Interface Local data structures Boundary conditions Independent paths Error-handling paths 382 PART THREE QUALIT Y AN D S ECURIT Y data structure, control flow, and data values just below, at, and just above maxima and minima are very likely to uncover errors. A good design anticipates error conditions and establishes error-handling paths to reroute or cleanly terminate processing when an error does occur. Yourdon [You75] calls this approach antibugging. Unfortunately, there is a tendency to incorporate error handling into software and then never test the error handling. Be sure that you design tests to execute every error-handling path. If you don’t, the path may fail when it is invoked, exacerbating an already dicey situation. Among the potential errors that should be tested when error handling is evaluated are: (1) error description is unintelligible, (2) error noted does not correspond to error encountered, (3) error condition causes system intervention prior to error handling, (4) exception-condition processing is incorrect, or (5) error description does not pro- vide enough information to assist in the location of the cause of the error. S afe H ome Designing Unique Tests The scene: Vinod’s cubical. Vinod: Those are okay, but I don’t see much The players: Vinod and Ed, members of the point in running both the 1234 and 6789 SafeHome software engineering team. inputs. They’re redundant... test the same thing, don’t they? The conversation: Ed: Well, they’re different values. Vinod: So these are the test cases you intend to run for the passwordValidation operation. Vinod: That’s true, but if 1234 doesn’t uncover an error... in other words... the password- Ed: Yeah, they should cover pretty much all Validation operation notes that it’s an invalid possibilities for the kinds of passwords a user password, it’s not likely that 6789 will show us might enter. anything new. Vinod: So let’s see... you note that the Ed: I see what you mean. correct password will be 8080, right? Vinod: I’m not trying to be picky here... it’s Ed: Uh-huh. just that we have limited time to do testing, so Vinod: And you specify passwords 1234 and it’s a good idea to run tests that have a high 6789 to test for error in recognizing invalid likelihood of finding new errors. passwords? Ed: Not a problem... I’ll give this a bit more Ed: Right, and I also test passwords that are thought. close to the correct password, see... 8081 and 8180. 19.3.1 Requirements and Use Cases In requirements engineering (Chapter 7) we suggested starting the requirements gath- ering process by working with the customers to generate user stories that developers can refine into formal use cases and analysis models. These use cases and models can be used to guide the systematic creation of test cases that do a good job of testing the functional requirements of each software component and provide good test cover- age overall [Gut15].  CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 383 The analysis artifacts do not provide much insight into the creation of test cases for many nonfunctional requirements (e.g., usability or reliability). This is where the customer’s acceptance statements included in the user stories can form the basis for writing test cases for the nonfunctional requirements associated with components. Test-case developers make use of additional information based on their professional experience to quantify acceptance criteria to make it testable. Testing nonfunctional requirements may require the use of integration testing methods (Chapter 20) or other specialized testing techniques (Chapter 21). The primary purpose of testing is to help developers discover defects that were previously unknown. Executing test cases that demonstrate the component is running correctly is often not good enough. As we mentioned earlier (Section 19.3), it is important to write test cases that exercise the error-handling capabilities of a compo- nent. But if we are to uncover new defects, it is also important to write test cases to test that a component does not do things it is not supposed to do (e.g., accessing privileged data sources without proper permissions). These may be stated formally as anti-requirements3 and may require specialized security testing techniques (Section 21.7) [Ale17]. These so-called negative test cases should be included to make sure the component behaves according to the customer’s expectations. 19.3.2 Traceability To ensure that the testing process is auditable, each test case needs to be traceable back to specific functional or nonfunctional requirements or anti-requirements. Often nonfunctional requirements need to be traceable to specific business or architectural requirements. Many agile developers resist the concept of traceability as an unneces- sary burden on developers. But many test process failures can be traced to missing traceability paths, inconsistent test data, or incomplete test coverage [Rem14]. Regres- sion testing (discussed in Section 20.3) requires retesting selected components that may be affected by changes made to other software components that it collaborates with. Although this is more often considered an issue in integration testing (Chapter 20), making sure that test cases are traceable to requirements is an important first step and needs to be done during component testing. 19.4 W h i t e -B ox T e s t i n g White-box testing, sometimes called glass-box testing or structural testing, is a test- case design philosophy that uses the control structure described as part of component- level design to derive test cases. Using white-box testing methods, you can derive test cases that (1) guarantee that all independent paths within a module have been exer- cised at least once, (2) exercise all logical decisions on their true and false sides, (3) execute all loops at their boundaries and within their operational bounds, and (4) exercise internal data structures to ensure their validity. 3 Anti-requirements are sometimes described during the creation of abuse cases that describe a user story from the perspective of a malicious user and are part of threat analysis (discussed in Chapter 18). 384 PART THREE QUALIT Y AN D S ECURIT Y 19.4.1 Basis Path Testing Basis path testing is a white-box testing technique first proposed by Tom McCabe [McC76]. The basis path method enables the test-case designer to derive a logical complexity measure of a procedural design and use this measure as a guide for defin- ing a basis set of execution paths. Test cases derived to exercise the basis set are guaranteed to execute every statement in the program at least one time during testing. Before the basis path method can be introduced, a simple notation for the repre- sentation of control flow, called a flow graph (or program graph), must be introduced.4 A flow graph should be drawn only when the logical structure of a component is complex. The flow graph allows you to trace program paths more readily. To illustrate the use of a flow graph, consider the procedural design representation in Figure 19.5a. Here, a flowchart is used to depict program control structure. Figure 19.5b maps the flowchart into a corresponding flow graph (assuming that no compound conditions are contained in the decision diamonds of the flowchart). Referring to Figure 19.5b, each circle, called a flow graph node, represents one or more procedural statements. A sequence of process boxes and a decision diamond can map into a single node. The arrows on the flow graph, called edges or links, represent flow of control and are analogous to flowchart arrows. An edge must terminate at a node, even if the node does not represent any procedural statements (e.g., see the flow graph symbol for the if-then-else construct). Areas bounded by edges and nodes are called regions. When counting regions, we include the area outside the graph as a region. An independent path is any path through the program that introduces at least one new set of processing statements or a new condition. When stated in terms of a flow Figure 19.5 (a) Flowchart 1 and (b) flow Edge graph 1 2,3 Node 2 6 R2 4,5 3 7 R3 8 6 9 R1 4 7 8 5 10 Region 9 11 10 R4 11 (a) (b) 4 In actuality, the basis path method can be conducted without the use of flow graphs. However, they serve as a useful notation for understanding control flow and illustrating the approach. CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 385 graph, an independent path must move along at least one edge that has not been traversed before the path is defined. For example, a set of independent paths for the flow graph illustrated in Figure 19.5b is Path 1: 1-11 Path 2: 1-2-3-4-5-10-1-11 Path 3: 1-2-3-6-8-9-10-1-11 Path 4: 1-2-3-6-7-9-10-1-11 Note that each new path introduces a new edge. The path 1-2-3-4-5-10-1-2-3-6-8-9-10-1-11 is not considered to be an independent path because it is simply a combination of already specified paths and does not traverse any new edges. Paths 1 through 4 constitute a basis set for the flow graph in Figure 19.5b. That is, if you can design tests to force execution of these paths (a basis set), every state- ment in the program will have been guaranteed to be executed at least one time and every condition will have been executed on its true and false sides. It should be noted that the basis set is not unique. In fact, a number of different basis sets can be derived for a given procedural design. How do you know how many paths to look for? The computation of cyclomatic complexity provides the answer. Cyclomatic complexity is a software metric that pro- vides a quantitative measure of the logical complexity of a program. When used in the context of the basis path testing method, the value computed for cyclomatic com- plexity defines the number of independent paths in the basis set of a program and provides you with an upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once. Cyclomatic complexity has a foundation in graph theory and provides you with an extremely useful software metric. Complexity is computed in one of three ways: 1. The number of regions of the flow graph corresponds to the cyclomatic complexity. 2. Cyclomatic complexity V(G) for a flow graph G is defined as V(G) = E − N + 2 where E is the number of flow graph edges and N is the number of flow graph nodes. 3. Cyclomatic complexity V(G) for a flow graph G is also defined as V(G) = P + 1 where P is the number of predicate nodes contained in the flow graph G. Referring once more to the flow graph in Figure 19.5b, the cyclomatic complexity can be computed using each of the algorithms just noted: 1. The flow graph has four regions. 2. V(G) = 11 edges − 9 nodes + 2 = 4. 3. V(G) = 3 predicate nodes + 1 = 4. 386 PART THREE QUALIT Y AN D S ECURIT Y Therefore, the cyclomatic complexity of the flow graph in Figure 19.5b is 4. More important, the value for V(G) provides you with an upper bound for the number of independent paths that form the basis set and, by implication, an upper bound on the number of tests that must be designed and executed to guarantee cover- age of all program statements. So in this case we would need to define at most four test cases to exercise each independent logic path. S afe H ome Using Cyclomatic Complexity The scene: Shakira’s cubicle. Shakira (exasperated): And exactly how do I The players: Vinod and Shakira—members of know which are the most error-prone? the SafeHome software engineering team who Vinod: V of G. are working on test planning for the security Shakira: Huh? function. Vinod: Cyclomatic complexity—V of G. Just The conversation: compute V(G) for each of the operations within each of the components and see which have Shakira: Look... I know that we should the highest values for V(G). They’re the ones unit-test all the components for the security that are most likely to be error-prone. function, but there are a lot of ‘em and if you consider the number of operations that have Shakira: And how do I compute V of G? to be exercised, I don’t know... maybe we Vinod: It’s really easy. Here’s a book that should forget white-box testing, integrate describes how to do it. everything, and start running black-box tests. Shakira (leafing through the pages): Okay, it Vinod: You figure we don’t have enough time doesn’t look hard. I’ll give it a try. The ops with to do component tests, exercise the opera- the highest V(G) will be the candidates for tions, and then integrate? white-box tests. Shakira: The deadline for the first increment Vinod: Just remember that there are no guar- is getting closer than I’d like... yeah, I’m antees. A component with a low V(G) can still concerned. be error-prone. Vinod: Why don’t you at least run white-box Shakira: Alright. But at least this’ll help me to tests on the operations that are likely to be the narrow down the number of components that most error-prone? have to undergo white-box testing. 19.4.2 Control Structure Testing The basis path testing technique described in Section 19.4.1 is one of a number of techniques for control structure testing. Although basis path testing is simple and highly effective, it is not sufficient in itself. In this section, other variations on control structure testing are discussed. These broaden testing coverage and improve the qual- ity of white-box testing. Condition testing [Tai89] is a test-case design method that exercises the logical conditions contained in a program module. Data flow testing [Fra93] selects test paths of a program according to the locations of definitions and uses of variables in the program.  CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 387 Figure 19.6 Classes of loops Simple loops Nested loops Loop testing is a white-box testing technique that focuses exclusively on the valid- ity of loop constructs. Two different classes of loops [Bei90] can be defined: simple loops and nested loops. (Figure 19.6). Simple Loops. The following set of tests can be applied to simple loops, where n is the maximum number of allowable passes through the loop. 1. Skip the loop entirely. 2. Only one pass through the loop. 3. Two passes through the loop. 4. m passes through the loop where m < n. 5. n − 1, n, n + 1 passes through the loop. Nested Loops. If we were to extend the test approach for simple loops to nested loops, the number of possible tests would grow geometrically as the level of nesting increases. This would result in an impractical number of tests. Beizer [Bei90] suggests an approach that will help to reduce the number of tests: 1. Start at the innermost loop. Set all other loops to minimum values. 2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter (e.g., loop counter) values. Add other tests for out-of-range or excluded values. 3. Work outward, conducting tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical” values. 4. Continue until all loops have been tested. 388 PART THREE QUALIT Y AN D S ECURIT Y 19.5 B lac k -B ox T e s t i n g Black-box testing, also called behavioral testing or functional testing, focuses on the functional requirements of the software. That is, black-box testing techniques enable you to derive sets of input conditions that will fully exercise all functional require- ments for a program. Black-box testing is not an alternative to white-box techniques. Rather, it is a complementary approach that is likely to uncover a different class of errors than white-box methods. Black-box testing attempts to find errors in the following categories: (1) incorrect or missing functions, (2) interface errors, (3) errors in data structures or external database access, (4) behavior or performance errors, and (5) initialization and termi- nation errors. Unlike white-box testing, which is performed early in the testing process, black-box testing tends to be applied during later stages of testing. Because black-box testing purposely disregards control structure, attention is focused on the information domain. Tests are designed to answer the following questions: ∙ How is functional validity tested? ∙ How are system behavior and performance tested? ∙ What classes of input will make good test cases? ∙ Is the system particularly sensitive to certain input values? ∙ How are the boundaries of a data class isolated? ∙ What data rates and data volume can the system tolerate? ∙ What effect will specific combinations of data have on system operation? By applying black-box techniques, you derive a set of test cases that satisfy the following criteria [Mye79]: test cases that reduce, by a count that is greater than one, the number of additional test cases that must be designed to achieve reasonable test- ing, and test cases that tell you something about the presence or absence of classes of errors, rather than an error associated only with the specific test at hand. 19.5.1 Interface Testing Interface testing is used to check that the program component accepts information passed to it in the proper order and data types and returns information in proper order and data format [Jan16]. Interface testing is often considered part of integration test- ing. Because most components are not stand-alone programs, it is important to make sure that when the component is integrated into the evolving program it will not break the build. This is where the use stubs and drivers (Section 19.2.1) become important to component testers. Stubs and drivers sometimes incorporate test cases to be passed to the component or accessed by the component. In other cases, debugging code may need to be inserted inside the component to check that data passed was received correctly (Section 19.3). In still other cases, the testing framework should contain code to check that data returned from the component is received correctly. Some agile developers prefer to do interface testing using a copy of the production version of the evolving program with some of this debugging code added. CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 389 19.5.2 Equivalence Partitioning Equivalence partitioning is a black-box testing method that divides the input domain of a program into classes of data from which test cases can be derived. An ideal test case single-handedly uncovers a class of errors (e.g., incorrect processing of all char- acter data) that might otherwise require many test cases to be executed before the general error is observed. Test-case design for equivalence partitioning is based on an evaluation of equiva- lence classes for an input condition. Using concepts introduced in the preceding sec- tion, if a set of objects can be linked by relationships that are symmetric, transitive, and reflexive, an equivalence class is present [Bei95]. An equivalence class represents a set of valid or invalid states for input conditions. Typically, an input condition is either a specific numeric value, a range of values, a set of related values, or a Boolean condition. Equivalence classes may be defined according to the following guidelines: 1. If an input condition specifies a range, one valid and two invalid equivalence classes are defined. 2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined. 3. If an input condition specifies a member of a set, one valid and one invalid equivalence class are defined. 4. If an input condition is Boolean, one valid and one invalid class are defined. By applying the guidelines for the derivation of equivalence classes, test cases for each input domain data item can be developed and executed. Test cases are selected so that the largest number of attributes of an equivalence class are exercised at once. 19.5.3 Boundary Value Analysis A greater number of errors occurs at the boundaries of the input domain rather than in the “center.” It is for this reason that boundary value analysis (BVA) has been developed as a testing technique. Boundary value analysis leads to a selection of test cases that exercise bounding values. Boundary value analysis is a test-case design technique that complements equiva- lence partitioning. Rather than selecting any element of an equivalence class, BVA leads to the selection of test cases at the “edges” of the class. Rather than focusing solely on input conditions, BVA derives test cases from the output domain as well [Mye79]. Guidelines for BVA are similar in many respects to those provided for equivalence partitioning: 1. If an input condition specifies a range bounded by values a and b, test cases should be designed with values a and b and just above and just below a and b. 2. If an input condition specifies a number of values, test cases should be developed that exercise the minimum and maximum numbers. Values just above and below minimum and maximum are also tested. 3. Apply guidelines 1 and 2 to output conditions. For example, assume that a temperature versus pressure table is required as output from an engineering 390 PART THREE QUALIT Y AN D S ECURIT Y analysis program. Test cases should be designed to create an output report that produces the maximum (and minimum) allowable number of table entries. 4. If internal program data structures have prescribed boundaries (e.g., a table has a defined limit of 100 entries), be certain to design a test case to exercise the data structure at its boundary. Most software engineers intuitively perform BVA to some degree. By applying these guidelines, boundary testing will be more complete, thereby having a higher likelihood for error detection. 19.6 O b j e c t - O r i e n t e d T e s t i n g When object-oriented software is considered, the concept of the unit changes. Encapsulation drives the definition of classes and objects. This means that each class and each instance of a class packages attributes (data) and the operations that manipulate these data. An encapsulated class is usually the focus of unit testing. However, operations (methods) within the class are the smallest testable units. Because a class can contain a number of different operations, and a particular oper- ation may exist as part of a number of different classes, the tactics applied to unit testing must change. You can no longer test a single operation in isolation (the conventional view of unit testing) but rather as part of a class. To illustrate, consider a class hierarchy in which an operation X is defined for the superclass and is inherited by a number of subclasses. Each subclass uses operation X, but it is applied within the context of the private attributes and operations that have been defined for the subclass. Because the context in which operation X is used varies in subtle ways, it is necessary to test operation X in the context of each of the subclasses. This means that testing operation X in a stand-alone fashion (the conventional unit-testing approach) is usually ineffec- tive in the object-oriented context. 19.6.1 Class Testing Class testing for object-oriented (OO) software is the equivalent of unit testing for conventional software. Unlike unit testing of conventional software, which tends to focus on the algorithmic detail of a module and the data that flow across the module interface, class testing for OO software is driven by the operations encapsulated by the class and the state behavior of the class. To provide brief illustrations of these methods, consider a banking application in which an Account class has the following operations: open(), setup(), deposit(), with- draw(), balance(), summarize(), creditLimit(), and close() [Kir94]. Each of these operations may be applied for Account, but certain constraints (e.g., the account must be opened before other operations can be applied and closed after all operations are completed) are implied by the nature of the problem. Even with these constraints, there are many permutations of the operations. The minimum behavioral life history of an instance of Account includes the following operations: open setup deposit withdraw close  CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 391 This represents the minimum test sequence for account. However, a wide variety of other behaviors may occur within this sequence: open setup deposit [deposit|withdraw|balance|summarize|creditLimit]n withdraw close A variety of different operation sequences can be generated randomly. For example: Test case r1: open setup deposit deposit balance summarize withdraw close Test case r2: open setup deposit withdraw deposit balance creditLimit withdraw close These and other random order tests are conducted to exercise different class instance life histories. Use of test equivalence partitioning (Section 19.5.2) can reduce the number of test cases required. S afe H ome Class Testing The scene: Shakira’s cubicle. Shakira: I know, I know. Here are some other The players: Jamie and Shakira, members of sequences I’ve come up with. (She shows the SafeHome software engineering team who Jamie the following sequences.) are working on test-case design for the #2: enable test [read]n test disable security function. #3: [read]n #4: enable disable [test | read] The conversation: Shakira: I’ve developed some tests for the Jamie: So let me see if I understand the intent Detector class [Figure 11.4]—you know, the one of these. #1 goes through a normal life history, that allows access to all of the Sensor objects sort of a conventional usage. #2 repeats the for the security function. You familiar with it? read operation n times, and that’s a likely sce- nario. #3 tries to read the sensor before it’s Jamie (laughing): Sure, it’s the one that been enabled... that should produce an error allowed you to add the “doggie angst” message of some kind, right? #4 enables and sensor. disables the sensor and then tries to read it. Shakira: The one and only. Anyway, it has an Isn’t that the same as test #2? interface with four ops: read(), enable(), dis- Shakira: Actually no. In #4, the sensor has able(), and test(). Before a sensor can be read, been enabled. What #4 really tests is whether it must be enabled. Once it’s enabled, it can be the disable op works as it should. A read() or read and tested. It can be disabled at any time, test() after disable() should generate the error except if an alarm condition is being pro- message. If it doesn’t, then we have an error in cessed. So I defined a simple test sequence the disable op. that will exercise its behavioral life history. (She shows Jamie the following sequence.) Jamie: Cool. Just remember that the four tests have to be applied for every sensor type since #1: enable test read disable all the ops may be subtly different depending Jamie: That’ll work, but you’ve got to do more on the type of sensor. testing than that! Shakira: Not to worry. That’s the plan. 392 PART THREE QUALIT Y AN D S ECURIT Y 19.6.2 Behavioral Testing The use of the state diagram as a model that represents the dynamic behavior of a class is discussed in Chapter 8. The state diagram for a class can be used to help derive a sequence of tests that will exercise the dynamic behavior of the class (and those classes that collaborate with it). Figure 19.7 [Kir94] illustrates a state diagram for the Account class discussed earlier. Referring to the figure, initial transitions move through the empty acct and setup acct states. The majority of all behavior for instances of the class occurs while in the working acct state. A final withdrawal and account closure cause the account class to make transitions to the nonworking acct and dead acct states, respectively. The tests to be designed should achieve coverage of every state. That is, the oper- ation sequences should cause the Account class to transition through all allowable states: Test case s1: open setupAccnt deposit (initial) withdraw (final) close Adding additional test sequences to the minimum sequence, Test case s2: o pen setupAccnt deposit(initial) deposit balance credit withdraw (final) close Test case s3: open setupAccnt deposit(initial) deposit withdraw accntInfo withdraw (final) close Figure 19.7 State diagram Setup Open Acct for the Empty Set up Account class acct acct Source: Kirani, Shekhar and Tsai, W. T., “Specification and Deposit (initial) Verification of Object-Oriented Programs,” Deposit Technical Report Working TR 94-64, acct University of Balance Minnesota, credit Withdraw December accntInfo 1994, 79. Withdrawal (Final) Close Dead Nonworking acct acct  CHAPTER 19 SOFT WA RE TESTING—COMPONENT LEVEL 393 Still more test cases could be derived to ensure that all behaviors for the class have been adequately exercised. In situations in which the class behavior results in a col- laboration with one or more classes, multiple state diagrams are used to track the behavioral flow of the system. The state model can be traversed in a “breadth-first” [McG94] manner. In this context, breadth-first implies that a test case exercises a single transition and that when a new transition is to be tested, only previously tested transitions are used. Consider a CreditCard object that is part of the banking system. The initial state of CreditCard is undefined (i.e., no credit card number has been provided). Upon reading the credit card during a sale, the object takes on a defined state; that is, the attributes card number and expiration date, along with bank-specific identifiers, are defined. The credit card is submitted when it is sent for authorization, and it is approved when authorization is received. The transition of CreditCard from one state to another can be tested by deriving test cases that cause the transition to occur. A breadth-first approach to this type of testing would not exercise submitted before it exercised undefined and defined. If it did, it would make use of transitions that had not been previously tested and would therefore violate the breadth-first criterion. 19.7 S u m m a ry Software testing accounts for the largest percentage of technical effort in the software process. Regardless of the type of software you build, a strategy for systematic test planning, execution, and control begins by considering small elements of the software and moves outward toward the program as a whole. The objective of software testing is to uncover errors. For conventional software, this objective is achieved through a series of test steps. Unit and integration tests (discussed in Chapter 20) concentrate on functional verification of a component and incorporation of components into the software architecture. The strategy for testing object-oriented software begins with tests that exercise the operations within a class and then moves to thread-based testing for integration (discussed in Section 20.4.1). Threads are sets of classes that respond to an input or event. Test cases should be traceable to software requirements. Each test step is accom- plished through a series of systematic test techniques that assist in the design of test cases. With each testing step, the level of abstraction with which software is consid- ered is broadened. The primary objective for test-case design is to derive a set of tests that have the highest likelihood for uncovering errors in software. To accomplish this objective, two different categories of test-case design techniques are used: white-box testing and black-box testing. White-box tests focus on the program control structure. Test cases are derived to ensure that all statements in the program have been executed at least once during test- ing and that all logical conditions have been exercised. Basis path testing, a white-box technique, makes use of program graphs (or graph matrices) to derive the set of linearly independent tests that will ensure coverage. Condition and data flow testing further exercise program logic, and loop testing complements other white-box techniques by providing a procedure for exercising loops of varying degrees of complexity. 394 PART THREE QUALIT Y AN D S ECURIT Y Black-box tests are designed to validate functional requirements without regard to the internal workings of a program. Black-box testing techniques focus on the infor- mation domain of the software, deriving test cases by partitioning the input and out- put domain of a program in a manner that provides thorough test coverage. Equivalence partitioning divides the input domain into classes of data that are likely to exercise a specific software function. Boundary value analysis probes the program’s ability to handle data at the limits of acceptability. Unlike testing (a systematic, planned activity), debugging can be viewed as an art. Beginning with a symptomatic indication of a problem, the debugging activity must track down the cause of an error. Testing can sometimes help find the root cause of the error. But often, the most valuable resource is the counsel of other members of the software engineering staff. Problems and Points to Ponder 19.1. Using your own words, describe the difference between verification and validation. Do both make use of test-case design methods and testing strategies? 19.2. List some problems that might be associated with the creation of an independent test group. Are an ITG and an SQA group made up of the same people? 19.3. Why is a highly coupled module difficult to unit test? 19.4. Is unit testing possible or even desirable in all circumstances? Provide examples to justify your answer. 19.5. Can you think of any additional testing objectives that are not discussed in Section 19.1.1? 19.6. Select a software component that you have designed and implemented recently. Design a set of test cases that will ensure that all statements have been executed using basis path testing. 19.7. Myers [Mye79] uses the following program as a self-assessment for your ability to spec- ify adequate testing: A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral. Develop a set of test cases that you feel will adequately test this program. 19.8. Design and implement the program (with error handling where appropriate) specified in Problem 19.7. Derive a flow graph for the program and apply basis path testing to develop test cases that will guarantee that all statements in the program have been tested. Execute the cases and show your results. 19.9. Give at least three examples in which black-box testing might give the impression that “everything’s OK,” while white-box tests might uncover an error. Give at least three examples in which white-box testing might give the impression that “everything’s OK,” while black-box tests might uncover an error. 19.10. In your own words, describe why the class is the smallest reasonable unit for testing within an OO system. Design element: Quick Look icon magnifying glass: © Roger Pressman  CHAPTER Software Testing— Integration Level 20 A single developer may be able to test software components without involving other team members. This is not true for integration testing where a component must interact properly with components developed by other team members. Inte- gration testing exposes many weaknesses of software development groups who have not gelled as a team. Integration testing presents an interesting dilemma for software engineers, who by their nature are constructive people. In fact, all test- ing requires that developers discard preconceived notions of the “correctness” of software just developed and instead work hard to design test cases to “break” the software. This means that team members need to be able to accept sugges- tions from other team members that their code is not behaving properly when it is tested as part of the latest software increment. Key artificial intelligence.................... 403 patterns.............................. 409 Concepts black-box testing...................... 397 regression testing..................... 402 bottom-up integration.................. 399 scenario-based testing................. 407 continuous integration.................. 400 smoke testing......................... 400 cluster testing......................... 404 thread-based testing................... 404 fault-based testing..................... 405 top-down integration................... 398 integration testing..................... 398 validation testing...................... 407 multiple-class partition testing........... 405 white-box testing...................... 397 Quick Look What is it? Integration testing assembles com- techniques and software requirements are ponents in a manner that allows the testing of exercised using “black-box” test-case design increasingly larger software functions with the techniques. intent of finding errors as the software is What is the work product? A set of test cases assembled. designed to exercise internal logic, interfaces, Who does it? During early stages of testing, a component collaborations, and external re­ software engineer performs all tests. How- quirements is designed and documented, ex- ever, as the testing process progresses, test- pected results are defined, and actual results ing specialists may become involved in are recorded. addition to other stakeholders. How do I ensure that I’ve done it right? Why is it important? Test cases must be de- When you begin testing, change your point signed using disciplined techniques to ensure of view. Try hard to “break” the software! that the components have been integrated Design test cases in a disciplined fashion, properly into the complete software product. and review the test cases you do create for What are the steps? Internal program logic is thoroughness. exercised using “white-box” test-case design 395 396 PART THREE QUALIT Y AN D S ECURIT Y Beizer [Bei90] describes a “software myth” that all testers face. He writes: “There’s a myth that if we were really good at programming, there would be no bugs to catch... There are bugs, the myth says, because we are bad at what we do; and if we are bad at it, we should feel guilty about it.” Should testing instill guilt? Is testing really destructive? The answer to these questions is, No! At the beginning of this book, we stressed the fact that software is only one element of a larger computer-based system. Ultimately, software is incorporated with other system elements (e.g., hardware, people, information), and systems testing (a series of system integration and validation tests) is conducted. These tests fall outside the scope of the software process and are not conducted solely by software engineers. However, steps taken during software design and testing can greatly improve the probability of successful software integration in the larger system. In this chapter, we discuss techniques for software integration testing strategies applicable to most software applications. Specialized software testing strategies are discussed in Chapter 21. 20.1 S o f t wa r e T e s t i n g F u n da m e n ta l s The goal of testing is to find errors, and a good test is one that has a high probabil- ity of finding an error. Kaner, Falk, and Nguyen [Kan93] suggest the following attri- butes of a “good” test: A good test has a high probability of finding an error. To achieve this goal, the tester must understand the software and attempt to develop a mental picture of how the software might fail. A good test is not redundant. Testing time and resources are limited. There is no point in conducting a test that has the same purpose as another test. Every test should have a different purpose (even if it is subtly different). A good test should be “best of breed” [Kan93]. In a group of tests that have a similar intent, time and resource limitations may dictate the execution of only those tests that have the highest likelihood of uncovering a whole class of errors. A good test should be neither too simple nor too complex. Although it is some- times possible to combine a series of tests into one test case, the possible side effects associated with this approach may mask errors. In general, each test should be executed separately. Any engineered product (and most other things) can be tested in one of two ways: (1) Knowing the specified function that a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function. (2) Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh,” that is, internal operations are performed according to specifications and all internal components have been adequately exercised. The first test approach CHAPTER 20 SOFT WA RE TESTING—INTEGRATION LEVEL 397 takes an external view of testing and is called black-box testing. The second requires an internal view of testing and is termed white-box testing.1 Both are useful in integration testing [Jan16]. 20.1.1 Black-Box Testing Black-box testing alludes to integration testing that is conducted by exercising the component interfaces with other components and with other systems. It examines some fundamental aspect of a system with little regard for the internal logical structure of the software. Instead, the focus is on ensuring the component executes correctly in the larger software build when the input data and software context specified by its preconditions is correct and behaves in the ways specified by its postconditions. It is of course important to make sure that the component behaves correctly when its preconditions are not satisfied (e.g., it can handle bad inputs without crashing). Black-box testing is based on the requirements specified in user stories (Chapter 7). Test-case authors do not need to wait for the component implementation code to be written once the component interface is defined. Several cooperating components may need to be written to implement the functionality defined by a single user story. Val- idation testing (Section 20.5) often defines black-box test cases in terms of the end- user visible input actions and observable output behaviors, without any knowledge of how the components themselves were implemented. 20.1.2 White-Box Testing White-box testing, sometimes called glass-box testing or structural testing, is an inte- gration testing philosophy that uses implementation knowledge of the control struc- tures described as part of component-level design to derive test cases. White-box testing of software is predicated on close examination of procedural implementation details and data structure implementation details. White-box tests can be designed only after component-level design (or source code) exists. The logical details of the program must be available. Logical paths through the software and collaborations between components are the focus of white-box integration testing. At first glance it would seem that very thorough white-box testing would lead to “100 percent correct programs.” All we need do is define all logical paths, develop test cases to exercise them, and evaluate results, that is, generate test cases to exercise program logic exhaustively. Unfortunately, exhaustive testing presents certain logisti- cal problems. For even small programs, the number of possible logical paths can be very large. White-box testing should not, however, be dismissed as impractical. Test- ers should select a reasonable number of important logical paths to exercise once component integration occurs. Important data structures should also be tested for validity after component integration. 1 The terms functional testing and structural testing are sometimes used in place of black-box and white-box testing, respectively. 398 PART THREE QUALIT Y AN D S ECURIT Y 20.2 I n t e g r at i o n T e s t i n g A neophyte in the software world might ask a seemingly legitimate question once all modules have been unit-tested: “If they all work individually, why do you doubt that they’ll work when we put them together?” The problem, of course, is “putting them together”—interfacing. Data can be lost across an interface; one component can have an inadvertent, adverse effect on another; subfunctions, when combined, may not produce the desired major function; individually acceptable imprecision may be mag- nified to unacceptable levels; and global data structures can present problems. Sadly, the list goes on and on. Integration testing is a systematic technique for constructing the software architec- ture while at the same time conducting tests to uncover errors associated with inter- facing. The objective is to take unit-tested components and build a program structure that has been dictated by design. There is often a tendency to attempt nonincremental integration, that is, to construct the program using a “big bang” approach. In the big bang approach, all components are combined in advance and the entire program is tested as a whole. Chaos usually results! Errors are encountered, but correction is difficult because isolation of causes is complicated by the vast expanse of the entire program. Taking the big bang approach to integration is a lazy strategy that is doomed to failure. Incremental integration is the antithesis of the big bang approach. The program is constructed and tested in small increments, where errors are easier to isolate and correct; interfaces are more likely to be tested completely; and a systematic test approach may be applied. Integrate incrementally and testing as you go is a more cost-effective strategy. We discuss several common incremental integration testing strategies in the remainder of this chapter. 20.2.1 Top-Down Integration Top-down integration testing is an incremental approach to construction of the soft- ware architecture. Modules (also referred to as components in this book) are integrated by moving downward through the control hierarchy, beginning with the main control module (main program). Modules subordinate (and ultimately subordinate) to the main control module are incorporated into the structure in either a depth-first or breadth- first manner. Referring to Figure 20.1, depth-first integration integrates all components on a major control path of the program structure. Selection of a major path is somewhat arbitrary and depends on application-specific characteristics (e.g., components needed to implement one use case). For example, selecting the left-hand path, components M1, M2, M5 would be integrated first. Next, M8 or (if necessary for proper function- ing of M2) M6 would be integrated. Then, the central and right-hand control paths are built. Breadth-first integration incorporates all components directly subordinate at each level, moving across the structure horizontally. From the figure, components M2, M3, and M4 would be integrated first. The next control level, M5, M6, and so on, follows. The integration process is performed in a series of five steps: 1. The main control module is used as a test driver, and stubs are substituted for all components directly subordinate to the main control module.  CHAPTER 20 SOFT WA RE TESTING—INTEGRATION LEVEL 399 Figure 20.1 Top-down M1 integration M2 M3 M4 M5 M6 M7 M8 2. Depending on the integration approach selected (i.e., depth or breadth first), subordinate stubs are replaced one at a time with actual components. 3. Tests are conducted as each component is integrated. 4. On completion of each set of tests, another stub is replaced with the real component. 5. Regression testing (discussed later in this section) may be conducted to ensure that new errors have not been introduced. The process continues from step 2 until the entire program structure is built. The top-down integration strategy verifies major control or decision points early in the test process. In a “well-factored” program structure, decision making occurs at upper levels in the hierarchy and is therefore encountered first. If major control prob- lems do exist, early recognition is essential. If depth-first integration is selected, a complete function of the software may be implemented and demonstrated. Early demonstration of functional capability is a confidence builder for all stakeholders. 20.2.2 Bottom-Up Integration Bottom-up integration testing, as its name implies, begins construction and testing with atomic modules (i.e., components at the lowest levels in the program structure). Bottom-up integration eliminates the need for complex stubs. Because components are integrated from the bottom up, the functionality provided by components subor- dinate to a given level is always available and the need for stubs is eliminated. A bottom-up integration strategy may be implemented with the following steps: 1. Low-level components are combined into clusters (sometimes called builds) that perform a specific software subfunction. 2. A driver (a control program for testing) is written to coordinate test-case input and output. 3. The cluster is tested. 4. Drivers are removed and clusters are combined, moving upward in the program structure. 400 PART THREE QUALIT Y AN D S ECURIT Y Figure 20.2 Bottom-up integration Mc Ma Mb D1 D2 D3 Cluster 3 Cluster 1 Cluster 2 Integration follows the pattern illustrated in Figure 20.2. Components are combined to form clusters 1, 2, and 3. Each of the clusters is tested using a driver (shown as a dashed block). Components in clusters 1 and 2 are subordinate to Ma. Drivers D1 and D2 are removed, and the clusters are interfaced directly to Ma. Similarly, driver D3 for cluster 3 is removed prior to integration with module Mb. Both Ma and Mb will ultimately be integrated with component Mc, and so forth. As integration moves upward, the need for separate test drivers lessens. In fact, if the top two levels of program structure are integrated top down, the number of driv- ers can be reduced substantially and integration of clusters is greatly simplified. 20.2.3 Continuous Integration Continuous integration is the practice of merging components into the evolving soft- ware increment once or more each day. This is a common practice for teams follow- ing agile development practices such as XP (Section 3.5.1) or DevOps (Section 3.5.2). Integration testing must take place quickly and efficiently if a team is attempting to always have a working program in place as part of continuous delivery. It is some- times hard to maintain systems with the use of continuous integration tools [Ste18]. Maintenance and continuous integration issues are discussed in more detail in Section 22.4. Smoke testing is an integration testing approach that can be used when product software is developed by an agile team using short increment build times. Smoke testing might be characterized as a rolling or continuous integration strategy. The software is rebuilt (with new components added) and smoke tested every day. It is designed as a pacing mechanism for time-critical projects, allowing the software team CHAPTER 20 SOFT WA RE TESTING—INTEGRATION LEVEL 401 to assess the project on a frequent basis. In essence, the smoke-testing approach encompasses the following activities: 1. Software components that have been translated into code are integrated into a build. A build includes all data files, libraries, reusable modules, and engineered components that are required to