Software Quality Assurance and Testing PDF

SOFTWARE QUALITY ASSURANCE AND TESTING SOFTWARE QUALITY ASSURANCE Dr. NGUYỄN QUANG VŨ E-mail: [email protected] Handphone: 0901.982.982 SOFTWARE QUALITY ASSURANCE OUTLINE Quality Assurance...

SOFTWARE QUALITY ASSURANCE AND TESTING SOFTWARE QUALITY ASSURANCE Dr. NGUYỄN QUANG VŨ E-mail: [email protected] Handphone: 0901.982.982 SOFTWARE QUALITY ASSURANCE OUTLINE Quality Assurance  An Umbrella Activity – Software Metrics  Software Standards Product Quality  Design Quality Metrics – Formal Approaches  Program Quality Metrics Project Quality  Reviews  Software Configuration Management Process Quality  ISO 9001/9000-3  The SEI Capability Maturity Model THE FOUR P’s IN SOFTWARE DEVELOPMENT set of activities How can measurement & we achieve Process feedback software quality? template software engineers set of artifacts participate result People Project Product education management testing & & & training monitoring measurement SOFTWARE QUALITY ASSURANCE (SQA) Quality assurance consists of those procedures, techniques, and tools applied by professionals to ensure that a product meets or exceeds pre-specified standards during it’s development cycle. E.H. Bersoff  It is an essential activity for any business that produces products used by others.  It needs to be planned and systematic. (It does not just happen)  It needs to be built into the development process. (A natural outcome of software engineering)  Continuous improvement is the overall goal. SQA — AN UMBRELLA ACTIVITY Methods and Tools Standards Testing and Procedures Quality Software Metrics Configuration and Management Measurement Formal Technical Reviews SQA — AN UMBRELLA ACTIVITY Includes all techniques used to improve the quality of the software: 1. methods and tools for System Requirements Capture, System Analysis, System Design, Implementation and Testing  to help ensure that we achieve a high quality system 2. standards and procedures compliance  to help ensure repeatability of the development process 3. metrics and measurement procedures  to help us improve the software development process 4. formal technical reviews at each step  to help uncover quality problems and to sign-off 5. software configuration management and change control  to help ensure changes are managed and controlled 6. multi-tiered testing  to help ensure effective error detection WHY SQA ACTIVITIES PAY OFF cost to find and fix a defect 100 80-130 log scale 13.00 10 SQA pays off here because defects are cheap to fix 4.00 2.00 1 1.00 1.30 Reqmts Design Code Test System Field test use SOFTWARE QUALITY What is software quality and how do we measure it? customer’s viewpoint meets specifications developer’s viewpoint easy to maintain, test,...  Software quality is not just about meeting specifications and removing defects! Other attributes of software that affect its quality: – safety – understandability – portability – security – testability – usability – reliability – adaptability – reusability – resilience – modularity – efficiency – robustness – complexity – learnability We need to select critical quality attributes early in the development process and plan for how to achieve them. PRINCIPLES OF SOFTWARE QUALITY ASSURANCE 1. We have a set of standards and quality attributes that a software product must meet.  There is a goal to achieve. key 2. We can measure the quality of a software product. point  There is a way to determine how well the product conforms to the standards and the quality attributes. 3. We track the values of the quality attributes.  It is possible to assess how well we are doing. 4. We use information about software quality to improve the quality of future software products.  There is feedback into the software development process. SOFTWARE STANDARDS Why are software standards important? 1. encapsulate best (or most appropriate) practices  acquired after much trial and error helps avoid previous mistakes 2. provide a framework around which to implement SQA process  ensures that best practices are properly followed 3. assist in ensuring continuity of project work  reduces learning effort when starting new work  product standards: define the characteristics all product artifacts should exhibit so as to have quality  process standards: define how the software process should be conducted to ensure quality software Each project needs to decide which standards should be: ignored; used as is; modified; created. SOFTWARE METRICS metric: any type of measurement that relates to a software system, process or related artifact control metrics - used to plan, manage and control the development process (e.g., effort expended, elapsed time, disk usage, etc.) predictor metrics - used to predict an associated product quality (e.g., cyclomatic complexity can predict ease of maintenance) external attribute: something we can only discover after the software has been put into use (e.g.,ease of maintenance) internal attribute: something we can measure directly from the software itself (e.g., cyclomatic complexity) SOFTWARE METRICS (cont’d) We want to use internal attributes to predict the value of external attributes. external internal maintainability # of parameters cyclomatic complexity reliability lines of code portability # of error messages usability length of user manual Problems: 1. hard to formulate and validate relationships between internal and external attributes 2. software metrics must be collected, calibrated, interpreted PRODUCT QUALITY: DESIGN QUALITY METRICS For a design component, the key quality attribute is maintainability. For design components maintainability is related to: – cohesion - how closely related is the functionality of the component? – coupling - how independent is the component? – understandability - how easy is it to understand what the component does? – adaptability - how easy is it to change the component? Problem: most of these cannot be measured directly, but it is reasonable to infer that there is a relationship between these attributes and the “complexity” of a component  measure complexity How? PRODUCT QUALITY: DESIGN QUALITY METRICS a) Structural fan-in/fan-out fan-in – number of calls to a component by other components fan-out – number of components called by a component  high fan-in => high coupling  high fan-out => calling component has high complexity b) Informational fan-in/fan-out – consider also the number of parameters passed plus access to shared data structures complexity = component-length x (fan-in x fan-out)2  It has been validated using the Unix system  It is a useful predictor of effort required for implementation PRODUCT QUALITY: DESIGN QUALITY METRICS c) IEEE Standard 982.1-1988  looks at: subsystem properties (number of subsystems and degree of coupling) database properties (number of attributes and classes)  compute a design structure quality index—DSQI (0-1)  used to compare with past designs; if DSQI is too low, further design work and review may be required  we can also consider changes made throughout the lifetime of the software and compute how stable the product is (i.e., how many changes have been made in subsystems in the current release)  define a software maturity index—SMI (0-1)  as SMI approaches 1, the product begins to stabilize  This slide is for information only – you are not required to know this! IEEE STANDARD 982.1-1988 S1 = the total number of subsystems defined in the program architecture S2 = the number of subsystems whose correct function depends on the source of data input or that produces data to be used elsewhere S3 = the number of subsystems whose correct function depends on prior processing S4 = the number of database items (includes data objects and all attributes that define objects) S5 = the total number of unique database items S6 = the number of database segments (different records or individual objects S7 = the number of subsystems with a single entry and exit  This slide is for information only – you are not required to know this! IEEE STANDARD 982.1-1988 Program structure: D1 = 1 if the architecture was developed using a distinct method D1 = 0 otherwise Subsystem independence: D2 = 1 - (S2/S1) Subsystems not dependent on prior processing: D3 = 1 - (S3/S1) Database size: D4 = 1 - (S5/S4) Database compartmentalization: D5 = 1 - (S6/S4) Subsystem entrance/exit characteristic: D6 = 1 - (S7/S1) DSQI = wiDi wi = relative weighting of each Di  This slide is for information only – you are not required to know this! IEEE STANDARD 982.1-1988 SMI = [MT - (Fa + Fc + Fd)] MT MT = the number of subsystems in the current release Fc = the number of subsystems in the current release that have been changed Fa = the number of subsystems in the current release that have been added Fd = the number of subsystems in the preceding release that were deleted in the current release PRODUCT QUALITY: PROGRAM QUALITY METRICS For an implementation component, the key quality attributes are reliability and difficulty of implementation. a) Halstead’s Software Science Looks at operators and operands in an implementation component and calculates values for component volume, V, component difficulty, D, and effort, E, required to implement the component. n1 = number of unique operators in a component n2 = number of unique operands in a component N1 = the total number of operators N2 = the total number of operands L = N1+ N2 (component length) V = L * log2(n1 + n2) (component volume in bits) D = (n1/2) * (N2/n2) (difficulty of implementing the component) E =V *D (effort required to implement the component) PRODUCT QUALITY: PROGRAM QUALITY METRICS b) McCabe’s Complexity Metric Looks at control flow in a component. Cyclomatic Complexity measures component’s logical complexity  an indication of the testing difficulty of a component Studies have shown a distinct relationship between the Cyclomatic Complexity of a component and the number of errors found in the source code, as well as the time required to find and fix errors. c) Other program quality metrics – length of code – length of identifiers – depth of conditional nesting Standards can be established to avoid complex components and/or highlight problem components. PRODUCT QUALITY: FORMAL APPROACHES a)Proving programs/specifications correct  logically prove that requirements have been correctly transformed into programs (e.g., prove assertions about programs) b)Statistical Quality Assurance  categorize and determine cause of software defects  80-20 rule 80% of defects can be traced to 20% of causes  isolate and correct 20% of the causes  effort directed to things that cause the majority of defects c) The Cleanroom Process  combination of above two approaches PROJECT QUALITY: REVIEWS The principal method of validating the quality of a project. requirements walkthroughs Requirements capture Leads to early analysis discovery of walkthroughs defects Analysis design walkthroughs Requirements, Analysis Design and Design introduce 50-60% code of all defects. walkthroughs Implementation test plan Formal technical reviews review can uncover 75% of these! Testing  10.2 SOFTWARE CONFIGURATION MANAGEMENT (SCM) An activity executed throughout the system life cycle to control change of products and life cycle artifacts. To what do we want to control changes? Software Configuration Management plans programs specifications documents/manuals procedures data identify support control audit report configuration item: an artifact to which we want to control changes  10.3.1 CONFIGURATION ITEM IDENTIFCATION AND DESCRIPTION each configuration item must be identified and described  a unique name This is  configuration item type metadata  project identifier about configuration  change and/or version information items  resources the configuration item provides or requires  pointer to actual configuration item the configuration items also must be organized (i.e., need to define the relationships that exist between configuration items)  object diagram domain model  domain model use-case model Define and construct a software library (database) that stores, manages and tracks configuration items.  10.3.2 CONTROLLING CONFIGURATION ITEMS: BASELINES A baseline is a time/phase in the software development (usually a project milestone) after which any changes must be formalized (e.g., go through a formal change control procedure).  In order to become a baseline, the configuration item must first pass a set of formal review procedures (e.g., formal code review, documentation review, etc.).  It then becomes part of the project software library.  After this a “check-out” procedure is applied to the item (i.e., access to and change of the configuration item is controlled)  Any modified configuration item must again go through a formal review process before it can replace the original (baselined) item. SCM BASELINES Requirements gathering Requirements Specification Analysis Analysis Specification Design Design Specification Implementation Source code, manuals Testing Test Plans/Procedures/Data Operational System CONTROLLING CONFIGURATION ITEMS: VERSION CONTROL  a configuration item usually evolves throughout the software engineering process (i.e., it will have several versions)  An evolution graph can be used to describe a configuration item’s change history version – configuration itemk is obtained by modifying configuration itemi – usually a result of bug fixes, enhanced system functionality, etc. – itemk supercedes original itemi; created in a linear order branch – a concurrent development path requiring independent configuration management variant – different configurations that are intended to coexist – e.g., different configurations depending on operating system type Oracle for Windows Oracle for Linux CONTROLLING CONFIGURATION ITEMS: VERSION CONTROL 1.0 branch line branch line 2.0 2.0.2.1 2.0.1 2.1 2.0.2.2 2.0.2 2.2 2.0.2.3 3.0 parallel developmen t mainline development SCM CHANGE CONTROL STOP  change request submitted by users/developers  change control authority evaluates, decides and issues an engineering Uncontrolled change order if approved  change configuration rapidly item is checked-out, changed, checked-in after SQA, leadscontrolled and version to chaos !  configuration item made available for use (promotion; release) SCM CHANGE CONTROL change request from user/developer change control authority evaluates and decides request is queued for action change request is denied assign individuals to configuration items user is informed “check-out” configuration items make changes review (audit) the change “check-in” the configuration items that have been changed establish a baseline for testing perform quality assurance and testing activities can be applied to “promote” changes for inclusion in next release internal requests rebuild appropriate version of software (i.e., user = developer) review (audit) the change to all configuration items and include changes in release external requests distribute the release (i.e., user = end-user) SCM AUDIT AND STATUS REPORTING Audit: ensures that changes have been properly implemented. – Have the proper steps and procedures been followed? checklist – Usually done by Quality Assurance (QA) group if SCM is a formal activity. Status reporting: keeps all parties informed and up-to-date on the status of a change. – A communication mechanism among project members to help keep them coordinated. – Can determine who made what changes, when and why.  10.3.5 SCM SUPPORT  a software library provides facilities to store, label, identify versions and track the status of the configuration items used by a everyday developer’s development single developer workspace check-in; check-out used by tracks master promotions other developers directory after SQA tracks for users software releases repository SCM BENEFITS  reduces the effort required to manage and effect change  improved productivity  leads to better software integrity and security  increased quality  generates information about the process  enhanced management control  maintains a software development database  better record keeping and tracking PROCESS QUALITY Does process quality product quality?  unlike manufactured products, software development has some unique factors that affect its quality: – software is designed, not manufactured – software development is creative, not mechanical – individual skills and experience have a significant influence – external factors (application novelty, commercial pressure)  software development processes are organization specific  people, technology may be more important than process  insufficient resources will always adversely affect quality PROCESS QUALITY: ISO 9001/ 9000-3 Focus is on generic ISO 9000 quality model process quality model management instantiated as customized for a particular Organization documents Organizational organization quality model quality process is used instantiated as to develop Project 1 Project 2 Project quality quality plan quality plan... management supports  certification is easy; can be a marketing ploy  This slide is for information only – you are not required to know this! ISO 9001/9000-3 STANDARD (1/3) A. Quality system framework 1. Management responsibility  quality policy defined, documented, understood, implemented and maintained  responsibilities and authorities for all personnel defined  in-house verification resources defined, trained and funded  a designated manager ensures that the quality program is implemented and maintained 2. Quality system  requires a documented quality system be established  should be an integral process throughout the entire life cycle 3. Internal quality system audits  requires audits be planned and performed  results communicated to management  deficiencies corrected 4. Corrective action  requires causes of non-conforming product be identified  potential causes of non-conformance eliminated  procedures changed as a result  This slide is for information only – you are not required to know this! ISO 9001/9000-3 STANDARD (2/3) B. Quality Life Cycle Activities 1. Contract review  includes planning design activities, identifying inputs and  contracts are reviewed to determine whether outputs, verifying the design and the requirements are adequately defined, agree controlling design changes with the bid, and can be implemented 6. Testing and validation  requires systems/products to be 2. Purchaser’s requirements specification tested and validated before use  procedures to identify and collect purchaser’s  records of testing are kept requirements 7. Acceptance  procedures for acceptance should be 3. Development planning established  production processes are defined and planned 8. Replication, delivery and 4. Quality record installation  procedures for the above should be  quality assurance process is planned established and maintained 5. Design and implementation 9. Maintenance  requires that maintenance activities  procedures to control and verify the design be performed as specified  This slide is for information only – you are not required to know this! ISO 9001/9000-3 STANDARD (3/3) C. Quality System Supporting Activities 1. Configuration management 6. Tools and Techniques  process of development and  are applied appropriately to support maintenance should be documented the development process and controlled 7. Purchasing 2. Document control  purchased products conform to their  distribution and modification of specified requirements documents should be controlled 8. Included software product 3. Quality records  should be verified and maintained  quality records should be collected, 9. Training maintained and dispositioned  training needs should be identified 4. Measurement and training provided since related  where appropriate, adequate tasks may require qualified personnel statistical techniques should be  training records should be identified and used to verify the maintained acceptability of process capability and product characteristics 5. Rules, practices and conventions  are in place and followed  12.3 PROCESS QUALITY: THE SEI CAPABILITY MATURITY MODEL (CMM) Intended to help a software organization improve their software development processes. Level 1 Organization: Initial process (ad hoc) – no formal procedures, no cost estimates, no project plans, no management mechanism to ensure procedures are followed Level 2 Organization: Repeatable process (intuitive) – basic project controls; intuitive methods used Focus is on Level 3 Organization: Defined process (qualitative) process – development process defined and institutionalized improvement Level 4 Organization: Managed process (quantitative) – measured process; process database established Level 5 Organization: Optimizing process – improvement feedback; rigorous defect-cause analysis and prevention  This slide is for information only – you are not required to know this! CAPABILITY MATURITY MODEL (CMM) Key Processes in Place Level 1: Initial process Level 4: Managed process Level 2: Repeatable process Quantitative Process Management Requirements Management Software Quality Management Software Project Planning Software Project Tracking & Oversight Level 5: Optimizing process Software Subcontract Management Fault Prevention Software Quality Assurance Technology Change Management Software Configuration Management Process Change Management Level 3: Defined process Organization Process Focus Organization Process Definition Training Program Integrated Software Management Software Product Engineering Intergroup Coordination Peer Reviews PROCESS QUALITY — NOTES  It is possible for a Level 1 organization to receive ISO 9000 certification!  A Level 3 organization would have little difficulty in obtaining ISO 9000 certification.  A Level 2 organization would have significant advantage in obtaining ISO 9000 certification.  excellent software developers (e.g., space shuttle) attain around level 3-4 only PEOPLE QUALITY: PEOPLE CAPABILITY MATURITY MODEL (PCMM) Intended to improve knowledge and skill of people. Level 1 – Initial Focus is on – no technical or management training provided; people staff talent not a critical resource; no organizational loyalty improvement Level 2 – Repeatable – focus on developing basic work practices; staff recruiting, growth and development important; training to fill skill “gaps”; performance evaluated Level 3 – Defined – focus on tailoring work practices to organization’s business; strategic plan to locate and develop required talent; skills-based compensation Level 4 – Managed – focus on increasing competence in critical skills; mentoring; team-building; quantitative competence goals; evaluation of effectiveness of work practices Level 5 – Optimizing – focus on improving team and individual skills; use of best practices SQA — THE BOTTOM LINE  an organization should have a quality manual which documents its quality assurance procedures  each project should have a quality plan which sets out the quality attributes that are most important for that project and how they will be assessed  an organization should have established standards for documentation  mechanisms (processes) should be established that monitor compliance with all quality requirements of the organization  reviews are the primary means of carrying out quality assurance  metrics can be used to highlight anomalous parts of the software which may have quality problems SUMMARY — SOFTWARE QUALITY Quality software does not just happen!  Quality assurance mechanisms should be built into the software development process  Developing quality software requires: – Management support and involvement – Gathering and use of software metrics – Policies and procedures that everyone follows – Commitment to following the policies and procedures even when things get rough! Testing is an important part of quality assurance, but its not all there is to obtaining a quality software product.