Principles of Software Testing

Principles of Testing Overview

• Professional testers are largely in agreement on the basic principles of testing, despite different terminology used throughout the industry.
• Proper testing is crucial, as the cost of errors can be substantial.

Testing Necessities

• Exhaustive testing will never be possible or practical.
• Fundamental test process includes industry standards.
• Determining expected results in advance of test execution is important.
• Re-testing and regression testing are critical for ensuring software quality before release.

Learning Objectives

• Basic testing terminology should be clear.
• Grasp the necessity of testing.
• Define error, fault, and failure distinctly.
• Recognize why errors happen and their potential costs.
• Recognize testing limitations and manage it as a risk.
• Grasp the fundamental test process.
• Understand the differing mindsets of developers and testers.
• Learn effective communication strategies for developers and testers.
• Understanding why testing of own work is ineffective.
• Knowing the importance of regression testing.
• Be able to define expected results in advance.
• Tests should be prioritized appropriately.

Testing Terminology

• A generally accepted set of testing definitions does not exist.
• The British Standard for Software Component Testing is a good source of testing definitions.
• It's important to ensure everyone understands terminology.

Why Testing is Needed

• Explores the reasons testing is necessary alongside the costs and consequences of software errors.

Errors, Faults, and Failures

• Error: A human action that produces an incorrect result.
• Fault: A manifestation of an error in software (also known as defects or bugs).
• Failure: Deviation of the software from its intended delivery or service, caused by a fault.

Reliability

• Reliability: The probability that software will perform without failure for a specified time and conditions.
• Includes MTBF, MTTF, and service level agreements.

Errors and How They Occur

• Human errors are unavoidable.
• Real-world pressures such as tight deadlines and budget restrictions intensify the risk of errors.

Cost of Errors

• The cost of errors ranges from negligible to massive financial losses or even loss of life.
• Nuclear reactor shutdown occurred due to a single line of incorrect code.
• Blue Cross of Wisconsin issued $60 million in unwarranted payments.
• Pepsi promotion error could've cost them $42 billion.
• Chemical Bank error caused $15 million to be incorrectly withdrawn.

London Ambulance Service (LAS) Failure

• The LAS new system failed, resulting in ambulances not arriving and loss of life on October 26th and 27th 1992.
• Problems: over-reliance on perfect information and difficulties in correcting system errors.
• Consequences: poor allocations, build-up of messages, slow-down, and telephone answering delays.

Exhaustive Testing

• Testing everything is impossible both theoretically and practically.
• Testing every possible scenario can be extremely time-consuming.

Testing and Risk

• Balance testing effort with the potential risk of failure.
• Prioritize tests based on risk assessment.
• Testing should focus on areas most likely to have defects.
• Execute the most important tests first.

Testing and Quality

• Testing improves software quality by identifying and removing faults.
• Testing measures software quality factors.

Testing and Requirements

• Testing may be determined by legal, contractual, regulatory, or mandatory requirements.

How Much Testing?

• Determining how much testing is enough involves balancing risks versus the cost of additional testing efforts.
• Thorough test planning and clear completion criteria are essential.
• Prioritising tests ensures critical functionalities are tested even if time runs short.

Fundamental Test Process

Introduction

• Testing must be planned.
• Good testing involves thinking out approaches.

Test Process Stages

• Test planning: Documents the overall approach, objectives, assumptions, and exceptions.
• Test specification: Designs test conditions and cases using recognized test techniques.
• Test execution: Running the tests on the system.
• Test recording: Keeping detailed records of test activities and results.
• Checking for test completion involves verifying if the completion criteria have been met and rerunning or designing some new test cases.

Successful Tests

• A successful test detects the faults.
• Finding faults early is more effective.

Completion Criteria

• Completion criteria determine when testing is complete, defined by cost, time, faults found, or coverage criteria.

Coverage Criteria

• Coverage criteria is in terms of items exercised by test suites.

Psychology of Testing

Purpose

• Aims to explore the differences in perspective between testers and developers.

Different Mindsets

• Testing finds faults, hence a destructive process.
• Development is creative.
• Developers aim to build new software that works the first time.
• Testers ensure the system meets user needs, is reliable, and functions correctly.
• Communication helps in getting bugs fixed.

Developer Communication

• Developers are creative and valued.
• Developers are often not good communicators; can specialize to only a few skills.

Tester Communication

• Testers are sometimes seen as destructive.
• Testers are multitasked.

Independence in Testing

• Independent testing is more effective.
• Test cases can be designed by different people.

Retesting and Regression Testing

• Re-testing is to ensure the fault is fixed.
• Regression testing verifies that modifications have not caused unintended adverse side effects.
• Maintain a regression test suite.
• Emergency fixes need a small regression test.

Expected Result

• Specifying expected results in advance of test execution is a must.
• Human error will cause more errors and testers might thing that a failure is a correct result

Some Testing Terminology

• Ad-hoc test care: tests executed without prior planning, when expected behavior is unknown.
• Bug report: tells someone that cares for follow-up when there is a discrepancy.
• Coincidental correctness: behavior appears correct, but it is coincidental.
• Failures: a product that doesn't behave correctly showing manifestation of a fault.
• Faults: the mistake in the code, the cause.
• Work-around: bypassing the error.
• Test Suite: Test cases that "adequately" test the product.
• Validation: Correspondance between the software and the specifications.

Programming Basics

• Write programs to tell the computer exactly what to do.
• Programs are sets of instructions for the computer.
• "Code" is another name for program.

Textual Programming

• Computers can't understand any language.
• Applications are programs.

Programming errors

• Bug is the program error.
• Debugging is fixing the application error.

Errors in Programming

• Syntax errors: are due to structure or grammar of the language (rules) applied.
• Routine errors: are due to non-existing situations like 1 divided by 0, is impossibility.
• Logical errors: are those in the meaning of the program.

The Cost of Bugs

• Costs of fixing bugs increase dramatically over time.
• Early detection is cheaper.

Why Bugs Happen

• Miscommunication.
• Software complexity.
• Programming errors.
• Changing requirements.