Software Quality Assurance Lecture 7 PDF

Lecture 7 Software Quality Prepare By Ts.Dr. Noorayisahbe Mohd Yaacob 1 Outline  Software Quality  Quality Assurance  Quality Control  Reliability Concepts 2 What is Software Quality Assurance?...

Lecture 7 Software Quality Prepare By Ts.Dr. Noorayisahbe Mohd Yaacob 1 Outline  Software Quality  Quality Assurance  Quality Control  Reliability Concepts 2 What is Software Quality Assurance? 3 What is Quality? Quality – developed product meets it’s specification Problems: Development organization has requirements exceeding customer's specifications (added cost of product development) Certain quality characteristics can not be specified in unambiguous terms (i.e. maintainability) Even if the product conforms to it’s specifications, users may not consider it to be a quality product (because users may not be involved in the development of the requirements) 4 Quality  Quality - "Fitness for purpose"  Who decides if your product has quality? ◼ Customer  Easy for observables ◼ e.g. quality of interaction, GUI, freedom from faults  Harder/impossible for other areas without assistance ◼ e.g. security, likelihood of continuing upgrades Quality  What do your customers want? ◼ They don't always know  Perhaps the wrong question ◼ Ask them what do they do  Typically they desire quality, they may not put it on a top 10 feature list ◼ quality = reliability, usability, consistency, etc.  Making quality software is about meeting a customer's needs - not necessarily what they say they want. What is Quality Management? Quality Management – ensuring that required level of product quality is achieved Defining procedures and standards Applying procedures and standards to the product and process Checking that procedures are followed Collecting and analyzing various quality data Problems: Intangible aspects of software quality can’t be standardized (i.e elegance and readability) 7 Software Quality Assurance 8 SQA  SQA is... ◼ Software - I hope you know what this is! ◼ Quality - "Fit for purpose" ◼ Assurance - to be assured, be given certainty, freedom from doubt, confidence, preventing defects  It's a collection of practices that reduce the risk of low quality products  More than just testing What are SQA, SQP, SQC, and SQM? SQA includes all 4 elements… Software Quality Assurance – is a systematic approach to ensuring that software products and processes meet specified quality standards and satisfy the needs of the users. Software Quality Planning – Selection of appropriate procedures and standards from this framework and adaptation of these to specific software project. A critical component of SQA, involves defining quality objectives, identifying standards, and outlining activities. 2. Software Quality Control – Definition and enactment of processes that ensure that project quality procedures and standards are being followed by the software development team. 3. Software Quality Metrics – Collecting and analyzing quality data to predict and control quality of the software product being developed. 10 SQA Cont. SQA Why are Standards Important? Standards provide encapsulation of best, or at least most appropriate, practice Standards provide a framework around which the quality assurance process may be implemented Standards assist in continuity of work when it’s carried out by different people throughout the software product lifecycle Standards should not be avoided. If they are too extensive for the task at hand, then they should be tailored. 13 SDS a Simplistic approach In most mature organizations: ISO is not the only source of SDS Process and Product standards are derived independently Product standards are not created by SQA 14 Quality Standards ISO - 9001 Elements  Quality System Requirements  Software Quality  Management Responsibility Responsibilities  Quality system  Management Responsibility  Contract review  Quality system  Design Control  Contract review  Document control  Design Control  Purchasing  Document control  Purchaser supplied product  Purchasing  Product identification and traceability  -  Process control  Product identification and traceability  Inspection and testing  Process control  Inspection, measuring and test equipment  Inspection and testing  Inspection and test status  -  Control of non-conforming product  Inspection and test status  Corrective action  -  Handling, storage, preservation, packaging  Corrective action and shipping  -  Quality records  Internal quality audits  Quality records  Training  Internal quality audits  Servicing  Training  Statistical techniques  -  Statistical techniques 16 Process and Product Quality Quality of development process directly affects the quality of delivered products. This is the factory approach. It doesn’t work because software is designed rather then manufactured. 17  Obtaining Quality  Quality control ◼ Activities that measure the quality of products produced ◼ E.g. Integration testing, code reviews  Quality Assurance ◼ Activities that focus on the process used to create the product ◼ E.g. Ensuring all testing is carried out at each level of the creation of software Software Quality Control 19 Methods of Software Quality Control SQC involves overseeing the software development process to ensure that the procedures and STD’s are being followed The following activities constitute SQC: Quality Reviews - in-process reviews of processes and products Reviews are the most widely used method of validating the quality of processes and products. Reviews make quality everyone's responsibility. Quality must be built-in. SQE is responsible for writing Quality Engineering Records (QERs) documenting their participation in these reviews. Tests - end-result verifications of products. These verifications are conducted after the software has been developed. Test procedures are followed during conduct of these activities. SQE is responsible for keeping the logs and some times for writing the test report. Quality Audits - in-process verifications of processes. These audits are conducted periodically (twice a month) to assess compliance to the process STD’s. 20 Quality Reviews Peer reviews - reviews of processes and products by groups of people. These reviews require pre-review preparation by all participants. If a participant is not prepared, then the review is not effective. This type of review requires participation of the SQE, moderator, recorder, author(s), and one or more critical reviewers. All issues found during these reviews are documented on AR forms. Walkthroughs - reviews of products by groups of people mostly without preparation. For example a requirements traceability review is a walkthrough. It involves tracing a requirement from customer requirements to the test procedures. All issues found during these reviews are documented on CAR forms. Desk inspections - reviews of products by individuals. These reviews involve people reviewing products by themselves (not in a group) and then submitting their comments to the author(s). The issues found during these reviews are treated in informal manner. 21 Tests Engineering Dry-run - test conducted by engineering without SQE. These tests include Unit Tests and engineering dry-runs of the formal tests. These engineering dry- runs are used to verify correctness and completeness of the test procedures. Also, these is the final engineering verification of the end-product before sell-off to SQE. All issues found during these tests are documented on STR forms. SQE Dry-run - test conducted by SQE. These tests are used to verify the end-product before the formal test with the customer. An SQE is sometimes responsible for writing the test report. However, if a separate test group is available, then SQE is relived of this obligation. All issues found during these tests are documented on STR forms. TFR - test conducted as “RFR - run-for-record” with the SQE and the customer. These tests include FAT and SAT. These tests are conducted to sell the end-product off to the customer. SQE is present at all such tests. All issues found during these tests are documented on STR forms. 22 Quality Audits SQE Audits - audits conducted by SQE to verify that the process STD’s are being followed. Examples of these audits are IPDS compliance, Configuration Control, and Software Engineering Management. All findings for these audits are documented on QER forms. The results of the audits are distributed to the next level of management (above project level). If the issue(s) are not fixed then the findings are elevated to upper management. Independent Audits - audits conducted by ISO generalists or other independent entities to verify that the process STD’s are being followed. These audits are usually conducted on a division/facility level. The results of these audits are distributed to upper management. 23 Defect Detection Formal bug finding activities include Quality Reviews and Tests sis sis aly aly An An ts e ts en tur en ap n ti m n m ca tio sig ire eC ire ign ra lifi qu De qu eg ua lin es Re Int Re ry se Q dD ina t Ba are are are es m ile m it T ste ftw ftw ftw de om ta eli So De Co Un So So Sy Pr Fr At Baseline Capture 0 System Requirements Analysis 0 79 D Software Requirements Analysis 0 0 1 E Preliminary Design 0 6 2 10 S T Detailed Design 1 0 0 0 42 T E Code 0 0 0 1 2 37 A C Unit Test 0 0 0 0 0 0 0 G T Software Integration 1 0 0 0 4 1 0 0 E E Software Qualification Test 0 0 0 0 0 0 0 0 0 D System Integration 1 0 0 0 4 5 0 0 0 System Test 0 0 0 0 0 0 0 0 Post System Test 0 0 0 0 0 0 0 0 0 % De fe cts Originate d in This Phase That We re Containe d By This Phase 93% 33% 91% 81% 86% % Defects Originated in This Phase Plus Defects 93% 11% 95% 79% 74% 0% 36% 0% That Escaped From Earlier Phases That Were Contained By This Phase % De fe cts Originate d In This Phase Out Of All De fe cts 44% 2% 6% 27% 22% 0% 0% 0% Chart Data Last Updated: 10/3/01 24 A Bug’s Life Postponed Rejected P X New Assigned Open Resolved Merged Tested Verified N A Engineer O R M T V SCCB Engineer Software Lead Tester SCCB Approves STR Accepts STR Resolves STR Verifies Fix Plans Merge Agrees Closure Integrator Duplicate Performs Merge D 25 Defect Prevention Defect Prevention – establishment of practices that lower the reliance on defect detection techniques to find majority of the bugs Lessons learned – learning from other peoples experiences and sharing own experiences with the other projects Managing With Metrics – collecting the metrics, understanding it, and making changes to the product or process based on analysis. Metrics must be standardized to be effective. Risk Analysis – identifying potential risks and opportunities early in the program and tracking them to realization. Build freeze – no changes are made to the code during formal tests. Unit-level testing guidelines – test plans and procedures for each UT Baseline acceptance criteria – establishment of closure criteria in advance (i.e. no P1 STRs at FAT TRR) 26 Continuity and Independence of SQA Software Quality Assurance team must be independent in order to take an objective view of the process and report problems to senior management directly If prescribed process is inappropriate for the type of software product which is being developed, then it should be tailored The standards must be upheld no matter how small the task. Prototyping doesn’t mean no standards. It means tailored standards. Quality is FREE, If it’s Everyone’s Responsibility! 27 Continuity and Independence of SQA Cont.  Preferable to have an SQA team to assist the Software Engineers doing quality assurance and quality control activities.  SQA team does not (necessarily) work on the project  SQA team is there to assist the Software Engineers in producing a high quality product. The SQA plan  Prepare an SQA plan for a project at the start of the project ◼ evaluations ◼ audits and reviews ◼ standards applicable ◼ procedures for error tracking/reporting ◼ documents produced by SQA group ◼ feedback provided to project team SQA Activities  Assists in selection of the process appropriate to the project  Reviews SE activities and work products to verify compliance with the defined software process  Ensures deviations in process and product are reported correctly using the procedure  Records any non-compliance ◼ Should it report to management? What is Reliability?  Reliability is a broad concept.  Reliability is one of the metrics that are used to measure quality.  It is a user-oriented quality factor relating to system operation. ◼ Intuitively, if the users of a system rarely experience failure, the system is considered to be more reliable than one that fails more often.  A system without faults is considered to be highly reliable. ◼ Constructing a correct system is a difficult task. ◼ Even an incorrect system may be considered to be reliable if the frequency of failure is “acceptable.”  Key concepts in discussing reliability: ◼ Fault ◼ Failure ◼ Time 31 What is Reliability?  Failure ◼ A failure is said to occur if the observable outcome of a program execution is different from the expected outcome.  Fault ◼ The adjudged cause of failure is called a fault. ◼ Example: A failure may be cause by a defective block of code.  Time ◼ Time is a key concept in the formulation of reliability. If the time gap between two successive failures is short, we say that the system is less reliable. ◼ Two forms of time are considered.  Execution time ()  Calendar time (t) 32 Definitions of Software Reliability  First definition ◼ Software reliability is defined as the probability of failure-free operation of a software system for a specified time in a specified environment.  Key elements of the above definition ◼ Probability of failure-free operation ◼ Length of time of failure-free operation ◼ A given execution environment  Example ◼ The probability that a PC in a store is up and running for eight hours without crash is 0.99.  Second definition ◼ Failure intensity is a measure of the reliability of a software system operating in a given environment.  Example: An air traffic control system fails once in two years. 33 Factors Influencing Software Reliability  A user’s perception of the reliability of a software depends upon two categories of information. ◼ The number of faults present in the software. ◼ The ways users operate the system.  This is known as the operational profile.  The fault count in a system is influenced by the following. ◼ Size and complexity of code ◼ Characteristics of the development process used ◼ Education, experience, and training of development personnel ◼ Operational environment 34 Q & A : Any question? 35