Software Quality Control Activities

Course Outline

• Part 1: Introduction to Software Quality
• Part 2: Software Quality Control Activities
• Part 3: Quality Metrics
• Part 4: Quality Management and Quality Systems
• Part 5: Software Quality Assurance Activities
• Part 6: Software Configuration Management

Quality Control Activities

• Find and correct defects as early as possible
• A defect is a non-conformance with functional, non-functional, or quality needs/requirements
• A corrective point of view should be taken

Validation and Verification

• Validation ensures that you built the right system and that the system meets users’ needs
• Verification ensures that you built the system right and that it satisfies stated requirements
• Quality control activity verifies or validates the system

Main Verification Methods

• Peer reviews
• Tests

Main Validation Methods

• Talking to users
• Simulations
• Prototype demonstrations
• Training materials piloting
• User tests

Software Quality Control Activities

• Static Controls
• Dynamic Controls

Quality Control Activities

• Static Controls
  • Manual
    • Reviews
    • Audits
  • Automatic
    • Automatic Static Analysis
      • Compilers
      • Flow analyzers
      • Symbolic execution
    • Formal Verification
• Dynamic Controls

Types of Reviews

• Desk Checking
• Informal Peer Review
• Walkthrough
• Inspection

Informal Review

• Participants in an informal review are authors or peers
• Responsibility is held by the authors
• Has the characteristics of; not planned, results that are not informed, and there is no follow-up of results

Formal Review

• Is a formal meeting that presents the current state of a project/product, with a structured analysis of results performed
• Properties include
  • At the end of development phases
  • Planned and public event
  • Results are reported
  • Participants are responsible for quality
  • Used to establish baselines
  • Technical + management (tracking progress, risks,...)

Formal Review Advantages:

• Milestones for the project
• Better preparation
• Provide reliable information to management
• Objectivity
• Sharing responsibility with authors

Formal Review Drawbacks:

• Impersonal
• Intimidating (round-robin)

Desk Checking

• Individual review activity
• Occurs as soon as development of an object is finished
• Performed by the author
• Objects can be informally exchanged
• It can be applied to any object

Informal Peer Reviews

• Review panel activity
• Samples are periodically reviewed
• It is more objective and effective
• Mainly applied to code

Inspection Process

• Planning: Involves Participants (3 to 8), Typical questions like (when, where, what, why…)
• Initial orientation or overview meeting: The author describes the important features of the product to be inspected
• Individual Preparation: Documentation associated to reviewed object and Checklists are created

Inspection Participants

• Coordinator or Moderator: Plans, Moderates the meeting, Writes a final report and Tracks action list
• Recorder or Secretary: Registers interesting elements and Helps producing final report
• Presenter or Author: Answers questions about the product and is a Reader
• Reader: Describes sections of the work product as the team proceed through the inspection
• Reviewers or Inspectors

Design Inspection Checklist

• Design Uniformity
• Interfaces among modules are defined correctly
• External Interfaces have been defined correctly
• Design covers all functions included in the requirements specification
• The design covers all non-functional requirements
• Design notation has been correctly applied
• Design Documentation is not ambiguous
• Design is detailed enough for its implementation in the selected language

About Checklists

• Not all defects are searched for
• Distributions of defect types are used to elaborate them and they should be changed over time
  • More frequent
  • More severe
  • More important
  • More costly
• Effectiveness of the inspection can be improved

Inspection process

• Inspection meeting
• Rework and Defect correction (author) with a Corrective actions report
• Follow-up (coordinator); all defects are corrected, new defects have not appeared, and Report to management

Inspection Meeting

• Procedure is object and/or checklist guided
• Focus is on efficiency to discover defects
• Results are an action list (authors), a summary report (management), and related issues report
• Possible ending situations:
  • Closure without defects
  • Closure after correction (minor defects)
  • A new inspection is needed (severe defects)

In-Class Group Exercise - Design Inspection

• Step 1: Individual Preparation (30 minutes Aprox.) with a list of potential defects delivered
• Step 2: Review Meeting (45 minutes Aprox.)
• Step 3: Review Report is delivered with Collected feedback (defects), Conclusions, Lessons Learned, Critics, and their Proposed Improved Checklist

Walkthrough

• A type of peer review based on a meeting that a guided tour of a product
• Only has two roles for participants: Presenter and Participant, with an optional Secretary

Walkthrough Process

• Planning phase
• Individual Preparation: Understand the product and Look for defects
• Walkthrough meeting: Guided by the author, Walk through various scenarios, goals for Educational value, to allow Defect detection and enable Discussion of alternatives

When to Apply a Walkthrough

• Not at the end of the phase, to allow room for changes
• Mainly for design, but also for requirements and code

In-Class Group Exercise - Design Walkthrough

• Step 1: Design (30 minutes)
• Step 2: Walkthrough
  • Define scenarios
  • Prepare slides
  • Choose presenter / secretary
• Step 3: Report, which includes: Collected feedback (defects, improvements), Conclusions, lessons learned, critics, Proposal of checklist for inspection

Audit Goals

• A research process aimed to
  • Verify conformance to procedures, instructions, specifications, codes, standards, or other applicable contractual requirements
  • Identify deviations or non-conformities
  • Propose solutions
  • Evaluate effectiveness and adequacy of implementation

Audit Differences With Revisions

• Why Audit: detect defects vs. Revisions to certify conformity
• When Auditing: Always in final states
• How Audit: Different procedure

Audit Types

• Product audits
  • Functional Audit
  • Physical Audit
  • In-process Audits
• Project audits
  • Team productivity and effectiveness
  • Methods and tools
  • Management
  • Root causes of problems
• Process audits
• Quality system audits

Product Audits - Functional

• Functional Product Audits
  • When they occur: After system tests
  • Inputs required: Test reports Requirements specification
  • Goal: To check that the necessary tests have been completed and that the product meets the requirements

Product Audits - Physical

• After functional audit
• Inputs includes product documentation (user manuals, installation, etc.) and a running product
• Goal is adequacy, completeness and precision of documentation
• When finished Product baseline is established and available for Release

Product Audits - In-Process

• Occur at the end of each phase and usually after a formal technical review
• Inputs include review reports
• Goal to check the adequacy of phase products
• When finished, the corresponding baseline is established and Phase ends

Audit Procedure

• Planning
  • Expected inputs and outputs
  • Output recipients
• Research
  • Opening session
  • Data collection
• Analysis
• Statistical analysis
• Evaluation in parallel
• Comparison of conclusions
• Study of causes
• Suggest solutions and improvements
• Results reporting

Test Our Knowledge

• To ensure that detail design is done according to the software development plan, a process audit should be used.

Automatic Static Analysis Tools

• Links to Automatic Static Analysis Tools are
  • HTTPS://DOCS.SONARQUBE.ORG/LATEST/
  • HTTPS://RULES.SONARSOURCE.COM/

Symbolic Execution Tools

• Link to Symbolic Execution Tools: HTTP://KLEE.GITHUB.IO/

Control Activities

• Static Controls
• Dynamic Controls
  • Black-box methods
    • Equivalence partitioning
    • Boundary values analysis
    • Cause-effect graphs
    • Error guessing
  • White-box methods
    • Coverage metrics
      • Sentences
      • Branching decisions
      • Branching conditions
      • Branching combination of conditions
      • Paths
    • Complexity metrics
      • Cyclomatic complexity
      • Essential Complexity
      • Real complexity

Dynamic Controls = Testing

• Test: Execution of a system with the goal of finding defects
• Test Design: Only the tests that reveal defects are worthy
• Process:
  • Detection
    • Run test
    • Find a failure
    • Find the defect that caused a failure
  • Correction
    • Find the way to solve it
    • Evaluate the effects of correction
    • Correct
• Cost:
  • Bigger for detection than correction
  • 50 - 60% of effort in a big project

Testing Levels

• Module or Unit Testing
• Integration Testing
  • Goal: Check interfaces among units
  • Strategies:
    • Bottom-up
    • Top-down
    • Big-bang
• System Testing
  • Meeting user requirements
• Acceptance Testing
  • Demonstrating system performance in real environment
• Regression Testing
  • In maintenance

V-Model for Testing

• Software Development:
  • Contract
  • User Specifications
  • Software Specifications
  • High Level Design
  • Detailed Design
  • Coding
• Quality Assurance:
  • Warranty
  • Acceptance Testing
  • System Testing
  • Integration Testing
  • Unit Testing
• Project Management

Test Our Knowledge

• The use of test automation provides the best return on investment for regression testing.

Test Design - Black-Box

• Also called Functional Test or Design-Oriented Test
• For every possible combination of input values
• Techniques for limiting the set of test cases includes:
  • Equivalence partitioning
  • Boundary values analysis
  • Cause-effect graphs
  • Error guessing
  • Scenario testing

Test Design - White-Box

• Also called Glass-Box or Structural Testing
• Every possible path in the control flow graph
• Techniques for evaluating the set of test cases:
  • Graph coverage metrics
    • Sentences
    • Branching decisions
    • Branching conditions
    • Branching combination of conditions
  • Complexity metrics

Test Our Knowledge

• Loop testing is included with white-box testing

Reliability Testing

• Software Reliability
  • “The probability of failure-free operation of a computer program for a specific time in a specific environment”
• Definition of an Operational Profile
  • Operations performed by the system
  • Actors initiating operations
  • Rate of occurrence and probability of occurrence of operations

Test Our Knowledge

• When software reliability measures are used to determine when to stop testing, the best types of test cases to use are those that: Exercise system functions in proportion to the frequency they will be used in the released product

Non-Functional Testing

• Usability testing
• Performance testing
• Security testing

Test Methodology

• Planning
  • Objects to test
  • Features to test
  • Resources
  • Schedule
  • Products to generate
  • Responsibilities
• Design
  • Methods to generate test cases
  • Exit criteria

Test Methodology

• Test case specification
  • Inputs
  • Expected exits
• Test procedure definition
  • Ordering of test cases
  • Preconditions for each test case
  • Termination conditions for each test case
• Execution
  • Report of results
• Analysis and evaluation
  • Goals reached