COMP 3521 chapter 8

Program Testing

• The purpose of testing is to demonstrate that a program functions as expected and to identify defects before release.
• Software testing involves executing a program with artificial data.
• Test results are checked for errors, anomalies, and information related to non-functional attributes.
• Testing reveals the presence of errors, but not their absence.
• Testing is a component of verification and validation, incorporating static validation techniques.

Program Testing Goals

• Software should meet requirements of both developer and customer.
• Custom software requires at least one test per requirement in the document.
• Generic software requires tests for all system features and their combinations in the product release.
• Testing aims to uncover incorrect, undesirable, or non-conforming behaviors in the software.
• Defect testing identifies system crashes, unwanted system interactions, incorrect computations, and data corruption.

Validation and Defect Testing

• Validation testing involves using test cases reflecting the system's expected use to ensure correct performance.
• Defect testing involves test cases designed to expose defects, which may be deliberately obscure and not reflective of normal system use.

Testing Process Goals

• Validation testing demonstrates that the software meets its requirements for both the developer and customer.
• Successful validation testing confirms that the system functions as intended.
• Defect testing identifies faults where the software's behavior is incorrect or doesn't match the specification.
• Successful defect testing causes the system to perform incorrectly, revealing defects.

Verification vs. Validation

• Verification checks if we are building the product correctly, ensuring software conforms to its specification.
• Validation checks if we are building the right product, ensuring the software meets user requirements.

Verification and Validation Confidence

• Verification and Validation (V&V) aims to establish confidence that the system is 'fit for purpose'.
• This depends on the system's purpose, user expectations, and the marketing environment.
• Confidence level depends on software criticality for the organization.
• User expectations vary depending on the software type.
• Early market entry may be prioritized over finding defects.

Inspections and Testing

• Software inspections analyze the static system representation to find issues, performing static verification.
• These may include tool-based document and code analysis.
• Testing involves exercising and observing product behaviour, known as dynamic verification.
• The system is run with test data and operational behaviour is watched.

Software Inspections

• Software inspections involve examining the source representation to discover anomalies and defects.
• Inspections do not require execution, and can be used before implementation.
• Inspections can be applied to any system representation including requirements, design, and test data.
• Inspections are effective in discovering program errors.

Advantages of Inspections

• Errors can mask other errors during testing, but inspections are static, eliminating concern for error interactions.
• Incomplete system versions can be inspected without extra costs.
• Program defects, compliance with standards, portability, and maintainability can be checked via inspection.

Inspections and Testing - Complementary Techniques

• They are not opposing verification techniques.
• Both should be used during the V & V process.
• Inspections check for conformance with the specification.
• Inspections cannot check non-functional characteristics like performance or usability.

Stages of Testing

• Development testing aims to discover bugs and defects during the software development.
• Release testing is performed by a separate team to test a complete system version before user release.
• User testing involves users or potential users testing the system in their own environment.

Development Testing

• Development testing are all testing activities by the system development team.
• Unit testing tests individual program units or object classes, focusing on object or method functionality.
• Component testing integrates individual units into composite components, focusing on component interfaces.
• System testing integrates some or all components, testing the system as a whole and focusing on component interactions.

Unit Testing

• Unit testing isolates individual components for comprehensive testing.
• It is a defect testing process.
• Units can be individual functions or methods, object classes with attributes and methods, and composite components with defined interfaces.

Object Class Testing

• Complete test coverage should include testing all operations, setting and interrogating attributes, and exercising the object in all possible states.
• Inheritance complicates object class tests because the information isn't localized.

Weather Station Testing

• Test cases should be defined for weather reports.
• Tests should cover weather calibration, startup, and shutdown.
• Using a state model, identify state transition and event sequences.
• Examples include Shutdown -> Running -> Shutdown, and Configuring -> Running -> Testing -> Transmitting -> Running.

Automated Testing

• Unit testing should be automated whenever possible, avoiding manual intervention.
• Automated unit testing uses testing frameworks like JUnit to write and run program tests.
• Unit testing frameworks offer generic test classes that are extended to create test cases, and report on the success or failure.

Automated Test Components

• The set-up initializes the system with inputs and expected outputs.
• The call part executes the object or method being tested.
• The assertion part compares the actual result to the expected result.

Choosing Unit Test Cases

• Test cases should verify that the component functions as expected.
• Test cases should reveal any component defects.
• Types include tests reflecting normal operation, and tests based on experience of where common problems arise.

Testing Strategies

• Partition testing identifies input groups with common characteristics processed in the same way.
• Tests should be selected from each of these groups.
• Guideline-based testing uses testing guidelines to choose test cases.
• These guidelines reflect past experiences on common programmer errors.

Partition Testing

• Input data and output results often fall into related classes.
• Each of these classes is an equivalence partition, where the program behaves equivalently for each member.
• Test cases should be chosen from each partition.

Testing Guidelines (Sequences)

• Test software with sequences containing only a single value.
• Use sequences of different lengths in different tests.
• Derive tests to access the first, middle, and last elements of a sequence.
• Test with sequences of zero length.

General Testing Guidelines

• Choose inputs to force error messages.
• Design inputs that overflow input buffers.
• Repeat inputs many times.
• Force invalid outputs.
• Force computation results to be too large or too small.

Component Testing

• Software components are often composite, made of interacting objects.
• For the weather station, the reconfiguration component involves objects dealing with each aspect of reconfiguration.
• Functionality of these objects is accessed through the component interface.
• Composite component testing focuses on the behavior of the component interface.
• Unit tests on individual objects within the component are presumed complete.

Interface Testing

• Objectives are to detect faults due to interface errors or invalid assumptions.
• Four types of interfaces: Parameter, Shared Memory, Procedural, Message Passing.

Interface Errors

• Interface misuse: A calling component makes an error in its interface use.
• Interface misunderstanding: A calling component assumes something about the called component’s behaviour that is wrong.
• Timing errors: The called and calling components operate at different speeds.

Interface Testing Guidelines

• Design tests with parameters at the extreme ends of their ranges.
• Always test pointer parameters with null pointers.
• Design tests that cause the component to fail.
• Use stress testing in message passing systems.
• Vary the order of component activation in shared memory systems.

System Testing

• Involves integrating components to test the overall system.
• The focus is on testing the interactions between components.
• It checks component compatibility, correct interaction, and proper data transfer.
• Emergent behavior of a system is tested.

System and Component Testing Integration

• Components, reusable components, and off-the-shelf systems can be integrated.
• Components from different team members may be integrated.
• System testing is a team effort rather than an individual one.
• Some companies may use a separate team without designer or programmer involvement for system testing.

Use-Case Testing

• Use-cases for identifying system interactions can be the basis of system testing.
• Each use-case involves several system components, forcing interactions to occur.
• Sequence diagrams document the components/interactions being tested.

Test Cases Derived from Sequence Diagram

• An input request should have an acknowledgement, and ultimately a report should be returned.
• Create summarized data to check the report is correctly organized.
• An input request to WeatherStation should generate a summarized report.
• Testing involves raw data to test SatComms and WeatherStation object, also used to test WeatherData object.

Testing Policies

• Exhaustive system testing is impossible, so policies define required system test coverage.
• Policies include: Testing all menu functions, testing combinations of functions accessed through the same menu and testing all functions with correct/incorrect user input.

Test-Driven Development (TDD)

• TDD interweaves testing and code development.
• Tests are written before the code, and ‘passing’ tests critically drives development.
• Code is developed incrementally with a test for each increment, progressing only when the code passes the test.
• Introduced in agile methods like extreme programming, but applicable in plan-driven processes.

TDD Process Activities

• Identify the increment of functionality, which should be small and implementable in a few code lines.
• Write a test for that function and implement as automated test.
• Run all tests, including the new one; initially, the new test will fail.
• Implement the functionality and rerun the tests.
• After all tests run successfully, the next part can be implemented.

Benefits of Test-Driven Development

• Every code segment has at least one associated test.
• A regression test suite is developed incrementally.
• Debugging is simplified because the problem source is obvious.
• The tests themselves act as documentation.

Regression Testing

• Checks that changes haven't ‘broken’ previously working code.
• Expensive in manual testing, but automated testing makes it simple.
• All tests must run ‘successfully’ before the change is committed.

Release Testing

• Involves testing a particular system release for use outside the development team.
• The primary goal is to ensure the system is good enough for use; it delivers specified functionality and does not fail during normal use.
• Tests are derived from the system specification.

Release Testing and System Testing - Key Differences

• A separate testing team should be responsible for release testing.
• System testing focuses in discovering bugs in the system (defect testing).
• Release testing is to check the system meets requirements and is good for external use (validation testing).

Requirements-Based Testing

• Examines each requirement and developing a test for it.
• Mentcare system requirements: if a patient is allergic to a medication, a warning must appear and if a prescriber ignores the warning they must provide a reason.

Requirements Tests

• Scenarios include: testing with no allergies, known allergies, multiple allergies and to check the specific warnings related to each case including the ability to overrule medication.

Features Tested by Scenario

• Authentication by logging on to the system
• Downloading and uploading of specified patient records to a laptop
• Home visit scheduling
• Encryption and decryption of patient records on a mobile device
• Record retrieval and modification
• Links with the drugs database
• System prompting function

Performance Testing

• Can test emergent properties like performance and reliability.
• Tests should reflect system use.
• Performance tests involve increasing load until system performance degrades.
• Stress testing deliberately overloads the system to test its failure behavior.

User Testing

• Users provide input and advice on system testing.
• User testing is essential because the influences from user’s environment cannot be tested.

Types of User Testing

• Alpha Testing: Users work with the development team to test the software.
• Beta Testing: Users experiment with the software and highlight any problems to the developers.
• Customers test a system to decide if it is ready to be accepted.

The Acceptance Testing Process

• Define acceptance criteria
• Plan acceptance testing
• Derive acceptance tests
• Run acceptance tests
• Negotiate test results
• Reject/accept the system

Stages in The Acceptance Testing Process

• Define acceptance criteria
• Plan acceptance testing
• Derive acceptance tests
• Run acceptance tests
• Negotiate test results
• Reject/accept system

Agile Methods and Acceptance Testing

• In agile methods, the user/customer is part of the team and decides on the system’s acceptability.
• Tests are defined by the user/customer and are integrated with other tests.
• There is no separate acceptance testing process.
• The embedded user might not be ‘typical’ and represent stakeholder interests.

Key points

• Testing shows the presence of errors but cannot prove their absence.
• Development testing is the software team's responsibility.
• A distinct team should be responsible for testing.
• Development testing includes unit, component, system testing.
• Break the software using test cases proven to discover defects in other systems.
• Tests should be automated, where possible.
• Test-first development involves writing tests before code.
• Scenario testing involves using a typical scenario to derive test cases.
• Acceptance testing assesses deployment readiness.