Principles of Software Testing

Principles of Testing

• Testing entails principles, lifecycle, static testing, dynamic test techniques, management, and tools.

Contents

• Testing is needed to identify errors, ensure reliability and fulfil expectations.
• Fundamental test process covers planning, specification, psychology, re-testing, expected results and test prioritisation.

Testing Terminology

• There isn't a universally accepted set of testing definitions.
• BS 7925-1 is a new standard.
• BS 7925-1 focuses on component testing and was developed by the BCS SIGIST.
• BS 7925-1 has been adopted by the ISEB / ISTQB.

Defining Bugs

• Error signifies a human action with an incorrect result.
• Fault represents an error's expression in software.
• Fault is also known as a defect or bug and can cause a failure if executed.
• Failure is when software deviates from expected delivery or service.
• Failure is an event; fault is a state of the software, caused by an error.

Reliability vs. Faults

• Reliability assesses the likelihood of software not causing system failure within a specific time and under defined conditions.
• Systems can be reliable even with faults.
• "Fault-free" software apps aren't always reliable.

Software Faults

• Software faults occur because software is written by humans beings.
• Faults occur more increasing pressure, to deliver to strict deadlines
• Faults may occur if you have ever written software.

Costs of Software Faults

• Software faults can result in huge sums, examples include: Ariane 5 ($7billion), Mariner space probe to Venus ($250m) and American Airlines ($50m).
• Incurring software faults can result in very little or nothing at all and minor inconvenience.
• Software is non-linear, small input, large effect.

Safety-Critical Systems

• Software faults can result in death or injury.
• Radiation treatment killed patients in Therac-25.
• Software faults led to train driver killed and aircraft crashes.
• Bank system overdraft letters caused suicide.

Testing Necessity

• Testing is necessary to highlight faults, assess software reliability, prevent expensive failures, avoid customer lawsuits and maintain business operations.

Exhaustive Testing

• System with 20 screens, average of 4 menus/screen, two input types, 10 fields/screen and 100 possible values -> 480,000 tests
• At one second per test, 480,000 tests will take 17.7 days.

Exhaustive Testing Defined

• Exhaustive testing involves testing all combinations of inputs and preconditions.
• Exhaustive testing takes an impractical amount of time.

Determining Enough Testing

• Enough testing depends on the risks for your system.

Testing Scope

• Testing scope depends on the risk of missing significant faults, incurring failure costs, or releasing untested, under-tested software.
• It's influenced by risks like losing credibility/market share, missing market windows, or over-testing.

Time Constraints on Testing

• Limited test time requires prioritizing tests based on risk such as: what to test first, what not to test and how thoroughly to test each item

Test Prioritization

• Prioritize tests so that if testing stops, the best possible testing has been completed in the given time.

Testing and Quality

• Testing measures software quality.
• When faults are found and removed, software quality increases.
• Testing tests system function, reliability, usability, maintainability, reusability, and testability.

Influences of Testing

• Testing influenced by contractual, legal, and industry-specific requirements.

Test Planning Levels

• Test Policy & Strategy = Company level
• High Level Test Plan = Project level (IEEE 829)
• Detailed Test Plan = Test stage level (IEEE 829)

Test Process

• The test process stages are planning, specification, execution, recording, and check completion.

Test Planning Particulars

• Test planning applies how the test strategy and project test plan will apply to the software under test.
• Test planning documents any exceptions to the test strategy.
• Test planning outlines software needed for tests, like stubs/drivers, and environment details.
• Test planning sets test completion criteria.

Test Specification

• Test specification involves identify conditions, design test cases and build tests.

What Is Good Test Case?

• A good test case finds faults, represents others, is easy to maintain and is cheap to use.

Breaking Down Effective Test Specification

• Identify: Determine 'what' to test and prioritize test conditions.
• Design: Determine 'how' to test 'what' which is to design test cases.
• Build: Implement the tests, data, scripts, etc.

Identifying Test Conditions

• List the conditions that we like to test and use the test design techniques specified in the test plan.
• Prioritize the test conditions, ensure most important conditions are covered.

Design Test Cases

• Design test input and test data by exercising one or more test conditions.
• Determine the results expected after predicting the outcome of each test case.
• Design sets of tests by using regression, building confidence and finding faults.

Build Test Cases

• Prepare test scripts.
• Less system knowledge requires more detailed scripts.
• Scripts for tools specify every detail.
• Prepare test data that must exist in files/databases at the start.
• Prepare expected results, defined before execution.

Test Execution

• Execute prescribed test cases.
• Execute the most important tests first
• Execution could be manual or automated.

Follow the Testing Plan

• Test records contain identities and versions of software under test and test specifications.
• Follow the plan to mark off progress on the test script.
• Document actual outcomes from the test and capture any other ideas you have for new test cases.
• Records are used to establish that all test activities have been carried out as specified.

Test Recording

• Testing compares actual outcome with expected outcome by logging discrepancies accordinly. Discrepancies may stem from a software fault, test fault, environment, version fault, or incorrect test run.
• Log coverage levels achieved for measures.
• After the fault has been fixed, repeat the required test activities, such as executing and designing a plan.

Check Test Completion Particulars

• Test completion criteria should be specified in the test plan.
• Repeat test activities if criteria not met to design more tests.

Test Completion Criteria Factors

• Completion criteria apply to all levels of testing to determine when to stop.
• Criteria include coverage, faults found, cost, and time.

Intellectual vs Clerical Testing

• Planning and Specification is more intelectual
• Execution and Recording is more clerical

Why Testing

• Testing is done to build confidence, demonstrate compliance with needs, identify faults and reduce costs.
• Testing meets user needs and assesses the software quality.

Traditional Testing Approaches

• A traditional testing approach shows that the system does what it should and doesn't do what it shouldn't.
• Goal of traditional testing is show working.
• Success is a system's works
• The outcome of traditional testing is faults left in.

Better Testing Approach

• To show system does what it shouldn't and doesn't do it should.
• The goal of this approach is to find faults
• Fastest achievement: difficult test cases

Testing Paradox

• The testing paradox is that testing is to find faults.
• Finding faults is to destroys confidence.
• Testing should build confidence.

Testers Role

• Testers bring bad news.
• Testers face time pressure.
• Testers need a different view or mindset.
• Testers communicate fault to authors and managers.

Rights of Testers

• Testers are entitled to have accurate info about progress and changes, insight, delivered code tested to a standard, be regarded as a professional, challenge specifications and have faults taken serious.
• Testers can make predictions about future fault levels and improve their testing process.

Responsibilities of Testers

• Testers are to follow the test plans, scripts, etc. as documented.
• Testers report faults objectively and factually and objectively assess risk.
• Testers prioritize what to report and communicate the truth.

Independence of Testing

• Testers can find 30%-50% of your own faults.
• Tester needs to take on assumptions and thought processes
• Tester see what you meant or want to see, not what is there + emotional attachment.

Levels of Independence

• None: tests designed by the person who wrote the software
• Tests designed by a different person
• Tests designed by someone from a different department or team (e.g. test team)
• Tests designed by someone from a different organisation (e.g. agency)
• Tests generated by a tool (low quality tests?)

Re-Testing

• Re-testing happens after faults are fixed.
• It involves running a test that failed and reported in a new software version.
• The test must be repeatable with same environment, versions, inputs and preconditions.
• After it passes, a fault has been fixed correctly.

Regression Test

• Regression aims to identify unexpected side-effects.
• It is also referred to as anti-regression or progression.
• Regression is a standard set of tests.
• It is worth automating at any level.
• Consider this a developing asset - must be maintained

When To Perform Regression Tests?

• Tests are performed after software changes, including faults fixed.
• Regression tests are ran when the environment changes.
• Regression tests are ran for emergency fixes.
• Regression tests evolve, run often, and may become large.

Maintenance

• To maintain, eliminate repetitive tests, combine test cases, select a subset of full regression suite and eliminate old fault fix tests.

Automating Regression Testing

• Test execution tools (e.g. capture replay) are regression testing tools.
• They re-execute tests which have already been executed.
• Once automated, regression tests can be run as often as desired.
• Automating tests is not trivial, taking 2-10x longer than manual runs.

Why Have Expected Results?

• Results should be predicted in advance as part of test design.
• 'Oracle Assumption' says correct outcome can be predicted.
• Predicted tests can help to prevent testing bias.

Prioritizing Tests

• Because everything cannot be tested.
• There is never enough time to do all the testing you would like..

Possible Criteria

• Ranking could be risk based such as severe failures, visible failure, etc.
• Ask customers their priorities.
• Areas most critical, changed, problems, complexity.

Key Points

• Testing is necessary because people make errors.
• Process entails planning, specification, execution, recording, checking completion.
• Independence and relationships are important in testing.
• Test fixes; regression test for the unexpected, complete tasks. Prioritize to do the best testing