Software Testing Chapter 6

Chapter 6 - Introduction to Software Testing

• Software testing aims to demonstrate that a program functions as intended and to find defects before deployment.
• Testing involves executing the program with artificial data to check results for errors, anomalies, and non-functional attributes.
• Testing reveals the presence of errors, but not their absence.
• Testing is part of verification and validation, including static validation techniques.

Program Testing Goals

• Demonstrate to developers and customers that the software meets requirements.
• For custom software, one test per requirement is needed in the requirement document. For generic software, tests for all features and combinations are needed.
• Discover situations in which software behavior is incorrect, undesirable, or doesn't match specifications.
• Defect testing focuses on undesirable behaviors like crashes, incorrect computations, and data corruption.

Validation and Defect Testing

• Validation testing ensures the system performs correctly based on expected use.
• Defect testing, also called failure-oriented testing, uses test cases to uncover defects and doesn't need to reflect normal system use, rather, it aims to expose failures.

Testing Process Goals

• Validation testing demonstrates software meets its requirements; a successful test indicates intended operation.
• Defect testing finds software faults where behavior is incorrect or doesn't meet specifications; a successful test reveals a system deficiency.

An Input-Output Model of Program Testing

• Input test data leads to output test results.
• Inputs can lead to anomalous system behavior, while outputs expose defects.

Verification vs. Validation

• Verification: "Are we building the product right?" Does the software conform to its specifications?
• Validation: "Are we building the right product?" Does the software meet the user's needs?

V & V Confidence

• V&V (Verification and Validation) aims to establish confidence that the system fits its purpose, depending on system purpose, user expectations, and marketing environment. Software criticality impacts confidence levels. User expectations may be low for certain kinds of software. Getting a product to market quickly sometimes outweighs finding defects.

What is a Fault?

• A fault (or bug) is an error in the software that causes unexpected or incorrect behavior.
• Types of faults: Syntax errors (in the code's grammar), Logic errors (miscalculations), Runtime errors (errors during execution)

Inspections and Testing

• Software inspections analyze static system representations (requirements, design) to identify problems. Often supplemented by tools analyzing documents and code.
• Software testing exercises and observes the system's operational behavior using test data.

Software Inspections

• Involve people examining the source code to detect anomalies and faults, without needing to run the system. Useful before implementation.
• Can be applied to any system representation (e.g., requirements, design, configuration data, testing data).
• Effective techniques for uncovering program errors.

Advantages of Inspections

• Errors can mask others during testing; inspections, being static, avoid these interactions.
• Incomplete systems can be inspected without extra costs (no need for test harnesses).
• Inspections can detect broader quality attributes, like compliance with standards, portability, and maintainability.
• Inspections and testing are complementary, not opposing techniques, and both are employed during the verification and validation (V&V) process.

A Model of the Software Testing Process

• A model illustrating the sequence: design test cases, prepare test data, run program, compare results, generate reports.

What is a Test Case?

• A specific set of instructions or conditions used to test a software application aspect. Identifies specific defects. Serves as documentation for future testing and debugging.
• Describes steps and expected behavior in a successful run.

What is a Test Plan?

• A document outlining the overall strategy and scope for testing a software project. Serves as a guide for the process. Includes tools, team members, steps, and deadlines to keep testing properly aligned with project deadlines.

Stages of Testing

• Development testing: Involves testing during development to identify and fix bugs or defects. (Unit, Integration, System)
• Release testing: A separate team tests a complete software version before delivery.
• User testing: Users or potential users test the software in their own environment. (Alpha, Beta, Acceptance)

Development Testing

• Unit testing: Assesses individual components or modules.
• Integration testing: Ensures modules work together well.
• System testing: Checks behavior of the entire system.

Release Testing

• Conducted by a separate team on a complete software version.
• Ensures all specified features, functionality, and performance requirements are upheld.

User Testing

• Done in the user's real-world environment; conducted by users or potential users.
• Alpha Testing: Testing at the developer's site, with user involvement.
• Beta Testing: Testing in the users' environment, with a nearly final product.
• Acceptance Testing: Users confirm if the product meets their needs.

General Testing Guidelines

• Provide inputs to create and check all error messages.
• Design inputs that cause overflow of input buffers.
• Repeat the same input or input series several times.
• Force invalid outputs from the system.
• Force computation outputs to be too large or too small.

Testing Policies

• Exhaustive system testing is impossible, so testing policies are developed to define desired test coverage.

Examples of testing policies

• Test all system functions accessed through menus.
• Test combinations of functions within the same menu (e.g., menu functions such as text formatting).
• If user input is necessary, then functions must tested with both correct and incorrect data.

Requirements-Based Testing

• Focuses on the system's requirements for each individual requirement to generate test cases to demonstrate expected behavior.

Key Points

• Testing can only show errors; it cannot prove absence of errors; this is often highlighted in a testing summary. System testing should be conducted by a group separate from the development team.
• Deliberately "break" the software using experience and guidelines, and implement automated tests wherever possible.