CS341 Software Quality Assurance & Testing Lecture Notes PDF

CS341: Introduction to Software Quality Assurance and Testing General Announcements ⬥ Reading materials ⬥ Lecture readings ⬥ Course Assessments Grades ⬥ Course Assessment Submissions ⬥ MST ⬥ Water bottle 2  “ Don’t document the problem,...

CS341: Introduction to Software Quality Assurance and Testing General Announcements ⬥ Reading materials ⬥ Lecture readings ⬥ Course Assessments Grades ⬥ Course Assessment Submissions ⬥ MST ⬥ Water bottle 2  “ Don’t document the problem, fix it. ~ Anonymous 3 1.0 Software Metric Goal-Based Software Measurement Framework Our Game-plan ⬥ Software measure classification ⬥ Goal-based paradigms: ⬦ Goal-Question-Metrics (GQM) ⬦ Goal-Question-Indicator-Metrics (GQIM) ⬥ Applications of GQM and GQIM 5 Goal-Based Measurement (GBM) ⬥ The primary question in goal-based measurement: ⬦ “What do we want to know or learn?” instead of “What metrics should we use?” ⬥ Because the answers depend on your goals, no fixed set of metrics is universally appropriate. ⬥ Instead of attempting to develop general-purpose measures, one has to describe an adaptable process that users can use to identify and define measures that provide insights into their own development problem. 6 GBM Process 1. Determining what to measure ⬦ Identifying entities ⬦ Classifying entries to be examined ⬦ Determining relevant goals 2. Determining how to measure ⬦ Inquire about metrics ⬦ Assign metrics 7 Identifying Entities ⬥ Product, process, resource [Fenton’ 91]. ⬥ Artifacts, activities, agents [Armitage’ 94]. ⬥ Both schemes are incomplete and haven’t enough representational power. ⬥ E.g., neither seems to deal with entities such as “defects”. ⬦ Is a defect a product, a process, or a resource? ⬦ Is defect an artifact, activity, or agent? 8 Process, Product, Resource ⬥ Process ⬦ A collection of software related activities usually associated with some timescale. ⬦ Different SE processes: development, maintenance, testing, reuse, configuration and management process, etc. ⬥ Product ⬦ Any artifacts, deliverables or documents that result from a process activity. ⬥ Resource ⬦ Entities required by a process activity. 9 Types of Attributes ⬥ Internal attributes ⬦ Attributes that can be measured entirely in terms of the process, product or resource itself separate from its behavior e.g., size ⬥ External attributes ⬦ Attributes that can be measured with respect to how the process, product or resource relates to its environment through its behavior e.g., quality 10 Sample Process Measures - 1 11 Sample Process Measures - 2 12 Sample Product Measures - 1 13 Sample Product Measures - 2 14 Sample Resource Measure 15 GQM Approach - 1 ⬥ Goal-Question-Metric (GQM) approach to process metrics provides a framework for deriving measures from organization or business goals. ⬥ References: ⬦ V.R. Basili and D. Weiss (1984), A Methodology for Collecting Valid Software Engineering Data, IEEE Trans. Software Engineering, vol. 10, pp.728-738. ⬦ V.R. Basili and H.D. Rombach (1988), The Tame Project: Towards Improvement- Oriented Software Environments, IEEE Trans. Software Engineering, vol.14, pp.758-773. 16 Basili’s GQM ⬥ Basili’s GQM process model [Basili 1992] 17 SEI’s GQM ⬥ The goal-driven process model defined by Software Engineering Institute (SEI) in1996 [Park 1996]. ⬥ This GQM process model defines ten steps to fulfill the task of developing a software measurement plan. ⬥ It focuses on the whole process of converting a business goal into several specific measurement goals and refining to a software measurement plan. ⬥ This process model is easy to understand and follow in that it works in a pipeline style, which means that the output of a step is the input of the next step [Chen 2003]. 18 GQM Approach - 2 ⬥ Goal: List major goals of development or maintenance project. ⬥ Question: Derive from each goal the questions that must be answered to determine if the goals are being met. ⬥ Metrics: Decide what must be measured in order to be able to answer the questions adequately. 19 GQM: Example - 1 20 GQM: Example - 2 ⬥ You are manager for a software development team and you have to decide upon the release time of your product. Construct a GQM tree related to this goal. 21 GQM: Example - 3 ⬥ Construct a GQM graph for the goal of “improving maintainability of your developed software”. 22 GQM: Data Analysis ⬥ Suppose that you propose to collect data to examine the effects of programmer experience on development effort. You are expecting that effort (staff-hours) will decrease as the experience of the people assigned increases (Fig. a) but the collected data shows effort increasing with increasing experience (Fig. b). What may be wrong? 23 Case Study: AT&T’s GQM ⬥ Goal: Better code base inspection ⬥ Subgoals: ⬦ Better inspection planning: [Plan] ⬦ Better monitor and control of the code: [Control] ⬦ Improving code inspection: [Improve] 24 AT&T’s GQM - 1 ⬥ GA: Code inspection: Plan ⬦ Q1: How much does the inspection process cost? ⬩ M1-1: Average effort per KLOC ⬩ M1-2: Percentage of re-inspection ⬦ Q2: How much calendar time does the inspection process take? ⬩ M2-1: Average effort per KLOC ⬩ M2-2: Total KLOC inspected 25 AT&T’s GQM - 2 ⬥ GB: Code inspection: Monitor & control ⬦ Q1: What is the quality of the inspection software? ⬩ M1-1: Average fault detected per KLOC ⬩ M1-2: Average inspection rate ⬩ M1-3: Average preparation rate ⬦ Q2: To what degree did the staff conform to procedure? ⬩ M2-1: Average inspection rate ⬩ M2-2: Average preparation rate ⬩ M2-3: Average lines of code inspected ⬩ M2-4: Percentage of re-inspection ⬦ Q3: What is the status of the inspection process? ⬩ M3: Total KLOC inspected 26 AT&T’s GQM - 3 ⬥ GC: Code inspection: Improve ⬦ Q1: How effective is the inspection process? ⬩ M1-1: Defect removal efficiency ⬩ M1-2: Average fault detected per KLOC ⬩ M1-3: Average inspection rate ⬩ M1-4: Average preparation rate ⬩ M1-5: Average lines of code inspected ⬦ Q2: What is the productivity of the inspection process? ⬩ M2-1: Average effort per fault detected ⬩ M2-2: Average inspection rate ⬩ M2-3: Average preparation rate ⬩ M2-4: Average lines of code inspected 27 Case Study: HP’s GQM ⬥ Three goals: ⬦ A: Maximize customer satisfaction ⬦ B: Minimize engineering effort and schedule ⬦ C: Minimize defects 28 Example: HP’s GQM-A - 1 ⬥ GA: Maximize customer satisfaction. ⬦ QA1: What are the attributes of customer satisfaction? ⬩ MA1: Functionality, usability, reliability, performance, supportability. ⬦ QA2: What are the key indicators of customer satisfaction? ⬩ MA2: Survey, quality function deployment (QFD). ⬦ QA3: What aspects result in customer satisfaction? ⬩ MA3: Survey, QFD. ⬦ QA4: How satisfied are customers? ⬩ MA4: Survey, interview record, number of customers severely affected by defects. ⬦ QA5: How many customers are affected by a problem? ⬩ MA5: Number of duplicate defects by severity 29 Example: HP’s GQM-A - 2 ⬥ GA: Maximize customer satisfaction. ⬦ QA6: How many problems are affecting the customer? ⬩ MA6-1: Incoming defect rate ⬩ MA6-2: Open critical and serious defects ⬩ MA6-3: Defect report/fix ratio ⬩ MA6-4: Post-release defect density ⬦ QA7: How long does it take to fix a problem? ⬩ MA7-1: Mean time to acknowledge problem ⬩ MA7-2: Mean time to deliver solution ⬩ MA7-3: Scheduled vs. actual delivery ⬩ MA7-4: Customer expectation of time to fix 30 Example: HP’s GQM-A - 2 ⬥ GA: Maximize customer satisfaction. ⬦ QA8: How does installing a fix affect the customer? ⬩ MA8-1: Time customers operation is down ⬩ MA8-2: Customers effort required during installation ⬦ QA9: Where are the bottlenecks? ⬩ MA9: Backlog status, time spent doing different activities 31 Example: HP’s GQM-B - 1 ⬥ GB: Minimize engineering effort & schedule. ⬦ QB1: Where are the worst rework loops in the process? ⬩ MB1: Person-months by product-component-activity. ⬦ QB2: What are the total life-cycle maintenance and support costs for the product? ⬩ MB2-1: Person-months by product-component-activity. ⬩ MB2-2: Person-months by corrective, adaptive, perfective maintenance. ⬦ QB3: What development methods affect maintenance costs? ⬩ MB3: Pre-release records of methods and post-release costs. 32 Example: HP’s GQM-B - 2 ⬥ GB: Minimize engineering effort & schedule. ⬦ QB4: How maintainable is the product as changes occur? ⬩ MB4-1: Incoming problem rate ⬩ MB4-2: defect density ⬩ MB4-3: Code stability ⬩ MB4-4: Complexity ⬩ MB4-5: Number of modules changed to fix one defect ⬦ QB5: What will process monitoring cost and where are the costs distributed? ⬩ MB5: Person-months and costs ⬦ QB6: What will maintenance requirements be? ⬩ MB6-1: Code stability, complexity, size ⬩ MB6-2: Pre-release defect density 33 Example: HP’s GQM-B - 3 ⬥ GB: Minimize engineering effort & schedule. ⬦ QB7: How can we predict cycle time, reliability, and effort? ⬩ MB7-1: Calendar time ⬩ MB7-2: Person-month ⬩ MB7-3: Defect density ⬩ MB7-4: Number of detects to fix ⬩ MB7-5: Defect report/fix ratio ⬩ MB7-6: Code stability ⬩ MB7-7: Complexity ⬩ MB7-8: Number of lines to change 34 Example: HP’s GQM-B - 4 ⬥ GB: Minimize engineering effort & schedule. ⬦ QB8: What practices yield best results? ⬩ MB8: Correlations between pre-release practices and customer satisfaction data ⬦ QB9: How much do the maintenance phase activities cost? ⬩ MB9: Personal-months and cost ⬦ QB10: What are major cost components? ⬩ MB10: Person-months by product-component-activity ⬦ QB11: How do costs change over time? ⬩ MB11: Track cost components over entity maintenance life- cycle. 35 Example: HP’s GQM-C - 1 ⬥ GC: Minimize defects ⬦ QC1: What are key Indicators of process health and how are we doing? ⬩ MC1: Release schedule met, trends of defect density, serious and critical defects. ⬦ QC2: What are high-leverage opportunities for preventive maintenance? ⬩ MC2-1: Defect categorization ⬩ MC2-2: Code stability ⬦ QC3: Are fixes effective with less side effects? ⬩ MC3: Defect report/fix ratio 36 Example: HP’s GQM-C - 2 ⬥ GC: Minimize defects ⬦ QC4: What is the post-release quality of each module? ⬩ MC4: Defect density, critical and serious detects. ⬦ QC5: What are we doing right? ⬩ MC5-1: Defect removal efficiency ⬩ MC5-2: Defect report/fix ratio ⬦ QC6: How do we know when to release? ⬩ MC6-1: Predicted defect detection and remaining defects. ⬩ MC6-2: Test coverage. 37 Example: HP’s GQM-C - 3 ⬥ GC: Minimize defects ⬦ QC7: How effective is the development process in preventing defects? ⬩ MC7: Post-release defect density ⬦ QC8: What can we predict will happen post-release based on pre-release data? ⬩ MC8: Corrections between pre-release complexity, defect density, stability, and customer survey data. ⬦ QC9: What defects are getting through and there causes? ⬩ MC9: Detect categorization 38 2.0 Software Metric Exercises Exercise - 1 ⬥ What different attributes might be measured by the followings: ⬦ The number of faults in program P using a set of test cases created specifically for the program P. ⬩ Failure (rate) intensity ⬦ The number of faults in program P using a standard set (benchmark) of test data. ⬩ Code efficiency ⬦ The number of faults in program P by programmer A during 1 hour. ⬩ (Individual) test efficiency 40 Exercise - 2 ⬥ A study of 1st grade school children found a high correlation between shoe size and reading ability, i.e., “the bigger the shoe size the higher reading ability”. Does this mean that shoe size is a good measure of intelligence of 1st grade school children? Use the concept of scale and measurement verification to explain why. ⬥ No. Shoe size correlates well with age. Age correlates well with reading ability. Neither would be a good measure, since neither would satisfy the conditions for measurement verification system. That is, the relationship between “shoe size” and “age” is not monotonically increasing for all members of population (e.g., some older students may have smaller shoe size) and also the relationship between “age” and “reading ability” is not monotonically increasing either (e.g., some younger kids may have higher reading ability). 41 Exercise 3 :GQM - 1 ⬥ Suppose that company X’s main business goal is to “find a way to deal with fragile customer requirements” – that is, customer requirements that may be incomplete or that may change during the software development process. For this project, we want to create a measurement plan using GQM technique. During the step of “identify what we want to know or learn” we must specify the followings: ⬦ Persons affected ⬦ Products and by-products ⬦ Inputs and resources ⬦ Internal artifacts ⬦ Activities and flow paths 42 Exercise 3 :GQM – 2 cont. ⬥ Ask at least one question for each of the (a) -(e) that will contribute to the goal. a) Persons affected: Who are the people that will be affected by this goal? b) Products and by-products: What are the products covered by this goal? c) Inputs and resources: What resources are potentially involved? d) Internal artifacts: What are the artifacts affected? e) Activities and flowpaths: What activities are involved? 43 Exercise 3 :GQM - 3 ⬥ Identify at least 3 relevant candidates for each of the (a) - (e) for this project. ⬥ Persons affected: Who are the people that will be affected by this goal? ⬦ Customers; Project Managers; Programmers; Document personnel; ⬦ Testers ⬥ Products and by-products: What are the products covered by this goal? ⬦ Documents; Working system; Budget ⬥ Inputs and resources: What resources are potentially involved? ⬦ People; Computers and development environment; Customers ⬥ Internal artifacts: What are the artifacts affected? ⬦ Prototype applications; Change requests; Project history and previous metrics ⬥ Activities and flow paths: What activities are involved? ⬦ Developing; Requirements gathering; Acceptance testing 44 Closing Remarks! ⬥ Metrics tell us whether a process is good enough to meet the customer's requirements or whether it needs to be better. 45 THANKS! ANY QUESTIONS? You can contact me at: ⬥ [email protected] ⬥ Please follow the email rule. 46