Software Testing and Testability

Testability

• Industry estimates state that 30-50% or more of system development costs go to testing
• Software testability relates to how easily software demonstrates its faults through testing via execution
• Testability is the likelihood that software fails on its next execution, assuming it has at least one fault
• If a fault is present, the goal is for it to fail during testing as quickly as possible
• Determining probability isn't easy and other measures will be used

Testing Model

• Testing involves a program, input, and output
• An oracle, whether human or mechanical, ascertains if the output aligns with the program's specification
• Output involves the produced value along with derived quality attribute measures like production time
• The program's internal state can be revealed to the oracle
• The oracle can determine if the program has entered an erroneous state

System Testability

• A system must control each component's inputs and observe their outputs
• This control/observation uses a test harness meaning specialized software or hardware to check the software being tested
• Test harnesses enable recording/playback of data over interfaces, and they can simulate external environments or aid in production
• Test harnesses help execute procedures and record output and may be substantial software, complete with architecture and stakeholders
• Testing is done by various developers, users, or QA personnel
• Portions of a system or the entire system may undergo testing
• The effectiveness in discovering faults and the rate at which tests achieve a desired coverage level is called response measures for testability
• Test cases stem from developers plus testing groups or customers, and may drive development via Agile

Netflix's Simian Army

• Netflix streams video and maintains a Simian Army made of tools for testing in its Amazon EC2 cloud service
• The Simian Army began with Chaos Monkey
• Chaos Monkey randomly terminates processes in the running system to test for failures and serious degradation
• The Latency Monkey induces artificial delays in client-server communication to simulate service degradation and measures upstream services response
• The Conformity Monkey detects and shuts down instances not adhering to best practices, or those instances not belonging to an autoscaling group
• The Doctor Monkey identifies unhealthy instances by using external health checks i.e CPU load
• The Janitor Monkey eliminates clutter and waste from Netflix cloud resources
• The Security Monkey finds and terminates security violations and ensures DRM and SSL certificates are valid
• The 10-18 Monkey finds configuration and runtime problems for customers in multiple regions, using different languages and character sets
• Netflix determines what faults to examine based on their impact and severity
• The Simian Army embodies the logging of operational data to reproduce faults and discover/log behavior in complex systems

Testability Scenario

• Code testing serves as validation to ensure an engineered artifact meets the stakeholder's needs and intended use
• Software architecture only involves running systems and source code, Architectural design reviews are another way of validation for architecture

Testability Scenario Portions

• Source of stimulus is from unit testers, integration testers, developers, and even end users on the customer side that may be automated
• Stimulus refers to tests executed upon completing a unit of coding or delivery to the customer
• Artifact refers to a unit of code, a subsystem, or the whole system being tested
• Environment includes the testing time (development, compile, deployment, runtime), or inside a test harness
• Response states the system can be controlled, tests can be done, and results found

Response Measures

• Response measures show how easily the system under test "gives up" its faults
• These include efforts to find faults and particular fault classes
• Finding a certain percentage of statements requires effort
• Longest test chain signifies test difficulty
• More include measures for effort, probability of more discovery, time frame, and time to prep the test environment
• Additional ones are measured by bringing the system brought into its intended state
• Error risk assessment and measuring risk reduction, in addition to rate the severity of the existing or found faults can also be used

Tactics for Testability

• Easing testing is the goal during software development

Categories of Tactics

• Enhance system traceability by adding system controllability and observability
• Lessen complexity within the system's design

Control and Observe System State

• Essential for control and observation that leads to defining testability
• Best way to control and observe is giving a software component inputs, and then observing its outputs

Tactics Specifics

• Software should keep state data
• Testers can assign values to data
• Accessible info to testers on request
• State data includes the running state, variable values, performance load, intermediate steps to help remake component behavior

Specialized Interfaces

• Interfaces that allow control or capture of a component's variable values, done either through a test harness or regular operation

Specialized Test Routines Examples

• A set/get method to set important attributes
• A report method to report the full state of the object
• A reset method to sets the internal state A method for verbose output, and event logging for performance

Test Interfaces/Methods

• Test interfaces/Methods separated so they can be removed when needed
• Performance/Safety critical systems may have issues fielding test code
• Code must behave the same way it was tested regardless of removals

Other Tactics

• Record/playback: used to re-create a fault when the state that caused it is often difficult to remake, and capturing is done at an interface
• Localize state storage: Convenient if a system, subsytem, or module to start in arbitrary test state if saved in one place
• Abstract Data Sources: Testing control by controlling the input data to test, interfaces more abstracted to insert test data easily
• Sandbox: Isolate the instance of system so its is unconstrained for testing, testing helped if operated with no permanent impact or has rollbacks

More Tactics

• Implementations have ability to no have consequences to an experiment, it's is used for scenario analysis, training, and simulation
• Virtualize resources: Build a controlled version of the resource, system clock and components can be tested at critical time boundaries (shift change)
• Executable Assertions: Design assertions to check values satisfy constraints and is placed at desired locations to indicate faulty state
• Assertions increase traceability when fail
• Type can be annotated with checking code
• Test cases have assertions to embed the test oracle in code, if correct and coded

Replacement

• Replacing barebones code for more for more elaborate software for testing
• Component Replacement swaps component and has more for testing
• Preprocessor Macros state reporting, activate probe statements, or return testing console
• Aspects report state

Limit Complexity

• Difficult to test complex software because of large operating state space
• Concerned about repeatable behavior to find the exact fault, not to just be able to make software fail
• Limit Complexity category has three tactics

Limit Structural Complexity

• Avoid/solve cyclic dependencies between components
• Isolate or encapuslate environment deps
• Reduce deps between components
• Limit inheritance hierarchy
• Limit polymorphism/dynamic calls
• Structural Metric, response of the class (count of methods of C plus methods of others by C), increasing traceability
• Increase cohesion/reduce coupling, all for improving traceability
• Controllability critical to making testing tractable

Less Complex

• Systems needing full data consistency all times more complex
• Build system under "eventual consistency" model for a simple to system design
• Architectures can help by lending traceability, layered style lets lower layers can be tested then higher layers
• Limit Non Determinism (the counterpart to limiting structural complexity)- harding to test system
• Tactic helps with finding all non determinism (parallelism) and weeding out what can be helped

Testability Summary

• System being testable makes it reliable
• Test Harness often used to execute tests
• System state helps control
• Ability to inject faults to system
• Different tactics helps support components
• Systems difficult cause too many connection and computations

Checklist

• Tasks that to execute test capture logging, state being controlled must be made
• Support must be added by systems of injection models
• Main abstractions tests if the code will perform correct of system operations
• Ensure data injection and setup are possible for test to be correct
• Testing should be performed on each possible structure

Additional Checklist

• Insure resources is available to have suites run and results happen the testing, limit access, capture fail data, inject limits (virtual tested should happen before to test correct

Testable Code

• Late bound code should be tested, with bindings and full support of the ability
• The goal should be reachable if is code tested
• Tech support is should be chosen to support all test goals

Test Automation

• Create useful test data set
• Capture sample from production and remove sensitive information
• Automation used more in more situations for less room of human error