Certified Tester Foundation Level Chapter 5

Dynamic Testing - White-Box

• White-box testing is a type of software testing that involves analyzing the internal structure of a component or system to detect faults.
• It is also known as structure-based, structure-oriented, or structural testing.

Structure-Based Testing

• Control flow-based testing:
  • Statement coverage (C0-coverage): all nodes of the control flow graph are executed.
  • Decision coverage (C1-coverage): all edges of the control flow graph are executed.
  • Boundary-interior coverage (Cgi): all loops are executed once, repeatedly, and rejected once.
  • Path coverage (C∞-coverage): all possible paths of the control flow graph are executed.
• Data flow-based testing:
  • All definitions (all defs): all definitions of variables are executed.
  • All definition-use pairs (all def-uses): all pairs of definitions and uses of variables are executed.
• Condition-based testing:
  • Simple condition coverage: all conditions are executed.
  • Multiple condition coverage: all combinations of conditions are executed.
  • Minimal multiple condition coverage: all minimal combinations of conditions are executed.

Experience-Based Testing

• Error guessing: testing based on the tester's experience and intuition about where errors might occur.
• Exploratory testing: testing that involves exploring the system's behavior without a preconceived notion of what might happen.

Choosing Test Design Techniques

• The choice of test design technique depends on the specific context and requirements of the system.
• Different techniques may be used for different parts of the system or for different testing phases.

White-Box Test Design Techniques

• Statement coverage: ensures that all statements in the code are executed.
• Decision coverage: ensures that all decisions in the code are executed.
• Condition coverage: ensures that all conditions in the code are executed.
• Path coverage: ensures that all possible paths of the code are executed.
• Mutation testing: involves modifying the code in small ways to see if the tests detect the changes.
• Data flow-based testing: ensures that all data flows through the code are executed.

Code Coverage

• Code coverage measures the percentage of the code that is executed during testing.
• It is used to evaluate the effectiveness of the testing.

Dynamic Test

• Dynamic test is a type of testing that involves executing the code to see how it behaves.
• It is used to evaluate the system's behavior under different conditions.

Black-Box vs. White-Box Testing

• Black-box testing: testing that involves only the input and output of the system, without knowledge of the internal structure.
• White-box testing: testing that involves analyzing the internal structure of the system to detect faults.

Test Design Techniques

• Test design techniques are used to derive and select test cases based on the analysis of the internal structure of the system.
• They are used to evaluate the completeness of the testing.

GCD Function

• The GCD (Greatest Common Divisor) function is an example of a function that can be tested using white-box testing.
• The function takes two integers as input and returns their greatest common divisor.

Control Flow Graph

• A control flow graph is a graphical representation of the flow of control through a program.
• It is used to visualize the structure of the program and to identify the different paths that the program can take.

UML/Java Specifications

• UML (Unified Modeling Language) is a language used to specify the behavior of systems.
• Java is a programming language used to implement the system.
• The specification of the GCD function in UML/Java is used to define the expected behavior of the function.### Control Flow-Based Coverage Criteria
• Statement coverage is a dynamic, control flow-based test design technique that requires all statements of the test object to be executed at least once.
  • Formula: Statement coverage = (Statements executed / Total statements) * 100%
  • Each test case is defined from a path of the control flow graph, and the edges of the graph that are on the path must be executed in the test case.

Example: Statement Coverage for gcd()

• Path: (1, 2-3, 4-6, 7, 8, 9-11, 8, 12, 13)
• gcd() function example is used to illustrate statement coverage.

Discussion about Statement Coverage

• Statement coverage is a relatively weak criteria.
• Empty edges have no meaning for statement coverage.
• 100% coverage of statements is not always reachable in practice.
• May indicate the presence of unreachable statements (dead code).

Decision Coverage

• Decision coverage is a white box test design technique that requires test cases to execute decision outcomes.
  • Formula: Decision coverage = (Decision results executed / Total decisions) * 100%
  • Each test case is determined by a path through the control flow graph to the decision being considered.
• Decision coverage is stronger than statement coverage.

Example: Decision Coverage for gcd()

• Path 1 = (1, 2-3, 4-6, 7, 8, 12, 13)
• Path 2 = (1, 2-3, 7, 8, 12, 13)
• Decision coverage is calculated based on the number of decision outcomes executed.

Discussion about Decision Coverage

• Decision coverage is a stronger criteria than statement coverage.
• Decision coverage can detect missing statements.
• 100% decision coverage implies 100% branch coverage and 100% statement coverage.

Addendum: Boundary-Interior Coverage

• Boundary-Interior coverage is a dynamic, control flow-based test design technique that requires each loop to be executed in at least three test cases.
  • Not at all (only if loops deter, e.g., while- or for-loop for which abort condition is true at first evaluation)
  • Exactly once
  • More than once
  • Formula: Boundary-Interior coverage = (Number of (gi)-tested loops executed / Total loops) * 100%

Addendum: Path Coverage

• Path coverage is a dynamic, control flow-based test design technique that requires each path in a control flow graph to be executed at least once.
  • Formula: Path coverage = (Number of paths executed / Total paths) * 100%
  • In practice, not reachable, more important as a theoretical measure.

Example: Path Coverage for gcd()

• Multiple paths are demonstrated for the gcd() function.

From Path to Test Case

• Previously, paths were determined through the control flow graph, which were to be executed by the test cases.
• Question: Which input values can enforce these paths?
• Idea: Consider the conditions of statements that determine control flow.
• Constraints for program variables can then be »computed«.

Test Case for Statement Coverage

• Logical (abstract) test case: { n > m ∧ n % m ≠ 0 ∧ n % (n % m) = 0 ; gcd(m, n) }

• Low-level (concrete) test case: { m = 4, n = 6; 2 }### Test Case for Decision Coverage

• Path: (1, 2-3, 7, 8, 12, 13)

• Logical (abstract) test case: {n ≤ m ∧ m % n = 0 ; gcd(m, n) }

• Low level (concrete) test case: { m = 4, n = 4; 4 }

Control Flow-Based and Object-Oriented Software

• Statement coverage and decision coverage are insufficient for object-oriented systems
• Complexity of object-oriented systems are often hidden in the correlations of classes or the interactions of objects
• Methods in the classes are usually less extensive and of less complexity (McCabe: 2-4)
• The required statement and/or decision coverage is then reachable with less effort
• Tool support for measuring coverage metrics is available for object-oriented systems

White-Box Techniques

• Exploratory testing, error guessing, experience-based testing, and evaluation of white-box techniques
• Power of white-box techniques, mutation testing, and concept of white-box techniques
• Further white-box test design techniques: mutation, LCSAJ, condition testing, data-flow-based testing
• Differences between black-box and white-box techniques, and evaluation of white-box techniques

Data Flow-Based Test

• Dynamic test in which the test cases are derived by considering the data used in the test object
• Completeness of the test (coverage) will be evaluated according to the data use
• Tests considering variable and data usage: value assignment, state change, use in expression, state preserving, and use in conditions
• Use in expression, state preserving: Definitional use, def (r), and computational use, c-use (m,n) and def (r)

Addendum: Data Flow-Based Coverage Criteria

• Hypothesis: defective data use!
• All-defs criteria: Use each definition at least once in one c-use or p-use without creating any new def in between
• Example: All-defs: 1, def m: p-use 3-4, 1, def n: p-use 3-4, ...
• Further criteria: All p-uses / All c-uses / All uses /...

Conditions in Programs (and Specifications)

• Truth values: false, true (often also 0, 1)
• Single (basic) condition: Type of variables boolean, operations with return value of type boolean, comparative operations
• Compounded condition: Linking partial conditions with boolean operators, basic-operators are and (∧, ∩), or (∨, ∪), not (¬ )
• Decisions often use compound conditions to control the program flow
• In Java: &, |, ^ Bit-wise and, or and exclusive-or-link, &&, || As above, but lazy evaluation