Software Testing Fundamentals

Software Testing (CS265) - W1-L2

• Software Testing (ST) is a procedure for identifying flaws and defects in software, aimed at creating high-quality, large-scale applications.
• The importance of ST is underscored by the significant costs of software failures highlighted by the Ariane-5 rocket failure (1996) and the RBS (2012) incident.
• These failures, along with the 1960s/1970s "Software Crisis," highlight software quality issues and the need for ST.
• Initial software development primarily focused on fixing bugs, but later included ST as a part of the creation process itself.
• Software defects can stem from various sources including mistakes by developers, code faults, and failures due to unanticipated or incorrect behavior under certain conditions.
• A significant percentage (around 50%) of software faults can be directly attributed to algorithmic or computational precision issues, according to HP Studies.
• Different types of software faults exist, including errors in logic, syntax, and documentation, as well as deficiencies in handling data or capacity.

Learning Objectives

• Understand the importance of Software Testing (ST) in software development.
• Distinguish between white-box and black-box testing approaches.
• Recognize that ST is not a perfect process.

Software Industry History

• IBM separated software and hardware in 1969.
• Personal computers (PCs) became mainstream in the 1980s.
• The World Wide Web emerged in the 1990s, followed by mobile devices in the 2000s.
• The software industry's overall value reached (almost) a trillion dollars by 2017.
• Significant financial losses (approximately $60 billion annually) are frequently attributed to inadequate software infrastructure.

Importance of Software Testing

• Various models and techniques are used for ST.
• ST is a crucial aspect of quality assurance and software engineering practices.
• The software industry has adopted and adapted various testing strategies in response to the software crisis.
• Defining roles, planning, documentation, and process management procedures are all critical components of ST.

Software Testing Definition

• Software Testing is a process or strategy to find, assess, and predict defects in software products.
• Its aim is to develop reliable, high-quality, large-scale applications.

Software Product Definition

• Software products consist of instructions in discrete programs designed to execute and deliver the required functionality.
• Data structures, configuration files, and documentation (system, user, and development) clarify functions, use, and design for the software.

Three Types of Defects in Software

• Mistakes (developer errors).
• Faults (implementation issues).
• Failures (undesired behavior).

Types of Software Faults

• Algorithmic issues: Incorrect mapping between inputs and outputs.
• Computational & precision errors: Calculated results not matching expected accuracy.
• Syntax errors: Violations of programming language rules.
• Documentation issues: Incompleteness or inaccuracies in documentation.
• Capacity/Boundary issues: Failures due to system overload or exceeding capacity/expected parameters.
• Degrees of Software Failure (Severity Level): • Level 1 (most severe): System crash, loss of function, no workaround, costly recovery • Level 2: Partial loss of function, data with possible workaround • Level 3: Loss of function or data, manual workaround possible • Level 4 (least severe): Cosmetic issues or minor inconvenience

Software Testing and Quality

• Complex code and sometimes vague requirements can contribute to reduced quality and an increased failure rate in software.
• Consequences of poor quality include damage to reputation, legal issues.

Why Software Testing

• Software testing reduces the risks and costs of software development.
• It allows software to be scaled effectively while maintaining quality.
• IT diminishes the likelihood of software product failures.
• Common signs of software failure include delays, cost overruns, failure to meet user needs, and quality issues in software functionalities

Need for Software Testing

• Difficulties in specifying requirements and producing consistent software behaviors contribute to software-related issues.
• Software development approaches include forward engineering and a feedback-based approach.

Two Engineering Approaches

• Forward Engineering: An ideal approach which involves steps of requirements, specification, design, and implementation followed by reliable development
• Feedback Approach: A practical approach using iterative design and implementation with testing and correction after each phase.

Validation in Development Process

• Ensure software aligns with users' needs to address validation aspects at different stages of Development

Specifications are Essential

• Software specifications provide a critical foundation for testing.
• Defining software behavior and expected errors is crucial.
• Specifications (including requirements) are expressed in English followed by structured formats (e.g. tables) for testing.

Example: Specification

• Discount programs based on customer points and gold membership status.

Example: Return Code

• Different return codes (e.g., "FULLPRICE," "DISCOUNT," "ERROR") depending on the points and membership status.

Example: How to Test?

• Example test cases for the discount program, including acceptable input values and expected outcomes.

Theory of Software Testing

• A theory of ST is needed to minimize the tests required without hindering thoroughness in covering all possible input data, producing results as defined in the specifications, and testing data selection with reliability and validity.

Feasibility of Exhaustive Testing

• Exhaustive testing is often impractical due to the extensive time needed to test all possible input values.

Example: Exhaustive Testing

• Identification of relevant input parameters and their possible values for exhaustive testing.

Example 2: Code

• Identification of the complexities of boundary conditions for bounding conditions related to testing input values.

Test Heuristics

• Develop effective testing strategies by employing proven heuristics with various testing types like random testing, white/black box, and other types.

Issues with Random Testing

• Oracle problem: Ensuring correctness of results
• Test data representation problem: Representative and balanced coverage to avoid issues of some input values being used too often or insufficiently.
• Tests completion problem: determining the sufficient number of tests needed for confidence.

Testing Types

• Black-box testing: Testing software functions without examining implementation details.
• White-box testing: Evaluating software's internals and the internal structures of its components to provide appropriate test cases.
• Experience-based testing: Evaluating software using expert insight and estimations to find bugs. (including faults insertion).

When to Finish Testing

• Testing should end based on budget parameters, completing planned test cases, or meeting predetermined failure rate criteria.

Static and Dynamic Testing

• Dynamic testing: Executing code and verifying its output.
• Static testing: Reviewing code without executing it.
• Static testing review-based techniques like pair programming, code reviews, and walkthroughs exist for improving the correctness of software processes.
• Mathematical techniques for program proving allow formal verification of correctness within the software components.

Testing in the Software Development Process

• A structured testing process within the stages of software development including unit, integration, system, sub-system, and acceptance testing stages.

ST Procedure

• Procedure of testing processes (ideal vs real) and expected vs actual results.

ST Procedure Example in Testing

• Example of an implementation using Java and TestNG for module verification.

ST Activities

• Clear description of the steps in software testing, encompassing analysis, identification, verification, implementation, execution, and examination of testing results.

ST Artefacts

• Detailed descriptions of the artifacts created throughout the software testing process including coverage items, test cases, test specifications, test code, test results, test procedure documents, and automated testing code.

Fault Models

• Describing how fault testing models help facilitate effective testing by matching the input to the fault models and expected outputs.
• Examples of fault models (e.g., hand axe, hammer) and how faults are linked to the ideal system.

Summary

• A comprehensive overview of software testing, including its purpose, importance, methodologies, and related concepts (mistakes, faults, failures), testing procedures, development process, and different testing models and types.

This quiz covers essential concepts in software testing, including types of flaws, severity levels of defects, and the impact of inadequate testing. Additionally, it explores the distinctions between white-box and black-box testing, as well as historical trends since the 1970s. Test your knowledge and understanding of crucial software testing principles.