Gift of Fire, Ch 8 Errors, Failures & Risks PDF

Gift of Fire, A: Social, Legal, and Ethical Issues for Computing Technology Fifth Edition Chapter 8 Errors, Failures and Risks...

Gift of Fire, A: Social, Legal, and Ethical Issues for Computing Technology Fifth Edition Chapter 8 Errors, Failures and Risks Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved What We Will Cover Failures and Errors in Computer Systems Case Study: The Therac-25 Increasing Reliability and Safety Dependence, Risk, and Progress Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (1 of 14) Most computer applications are so complex it is virtually impossible to produce programs with no errors The cause of failure is often more than one factor Computer professionals must study failures to learn how to avoid them Computer professionals must study failures to understand the impacts of poor work Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (2 of 14) Problems for Individuals Billing errors Inaccurate and misinterpreted data in databases – Large population where people may share names – Automated processing may not be able to recognize special cases – Overconfidence in the accuracy of data – Errors in data entry – Lack of accountability for errors Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (3 of 14) System Failures Galaxy IV Amtrak Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (4 of 14) System Failures Voting systems – Technical failures – Programmers or hackers rigging software to produce inaccurate results. – Vulnerability to viruses Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (5 of 14) System Failures Denver Airport – Baggage system failed due to real world problems, problems in other systems and software errors – Main causes: ▪ Time allowed for development was insufficient ▪ Denver made significant changes in specifications after the project began Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (6 of 14) System Failures Airports in Hong Kong and Kuala Lumpur – Comprehensive systems failed because designers did not adequately consider potential for user input error. Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (7 of 14) System Failures Abandoned systems – Some flaws in systems are so extreme that the systems are discarded after wasting millions, or even billions, of dollars. Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (8 of 14) System Failures Lack of clear, well-thought-out goals and specifications Poor management and poor communication among customers, designers, programmers, etc. Institutional and political pressures that encourage unrealistically low bids, low budget requests, and underestimates of time requirements Use of very new technology, with unknown reliability and problems Refusal to recognize or admit a project is in trouble Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (9 of 14) System Failures Legacy systems – Reliable but inflexible – Expensive to replace – Little or no documentation Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (10 of 14) What Goes Wrong? The job they are doing is inherently difficult. Sometimes the job is done poorly. Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (11 of 14) What Goes Wrong? Design and development problems – Inadequate attention to potential safety risks – Interaction with physical devices that do not work as expected – Incompatibility of software and hardware, or of application software and the operating system – Not planning and designing for unexpected inputs or circumstances – Confusing user interfaces – Insufficient testing – Reuse of software from another system without adequate checking – Overconfidence in software – Carelessness Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (12 of 14) What Goes Wrong? Management and use problems – Data-entry errors – Inadequate training of users – Errors in interpreting results or output – Failure to keep information in databases up to date – Overconfidence in software by users Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (13 of 14) What Goes Wrong? Misrepresentation, hiding problems and inadequate response to reported problems Insufficient market or legal incentives to do a better job Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Failures and Errors in Computer Systems (14 of 14) What Goes Wrong? Reuse of software: the Ariane 5 rocket and “No Fly” lists – It is essential to reexamine the specifications and design of the software, consider implications and risks for the new environment, and retest the software for the new use. Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Case Study: The Therac-25 (1 of 6) Therac-25 Radiation Overdoses Massive overdoses of radiation were given; the machine said no dose had been administered at all Caused severe and painful injuries and the death of three patients Important to study to avoid repeating errors Manufacturer, computer programmer, and hospitals/clinics all have some responsibility Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Case Study: The Therac-25 (2 of 6) Software and Design problems Re-used software from older systems, unaware of bugs in previous software Weaknesses in design of operator interface Inadequate test plan Bugs in software – Allowed beam to deploy when table not in proper position – Ignored changes and corrections operators made at console Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Case Study: The Therac-25 (3 of 6) Why So Many Incidents? Hospitals had never seen such massive overdoses before, were unsure of the cause Manufacturer said the machine could not have caused the overdoses and no other incidents had been reported (which was untrue) The manufacturer made changes to the turntable and claimed they had improved safety after the second accident. The changes did not correct any of the causes identified later. Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Case Study: The Therac-25 (4 of 6) Why So Many Incidents? Recommendations were made for further changes to enhance safety; the manufacturer did not implement them. The FDA declared the machine defective after the fifth accident. The sixth accident occurred while the FDA was negotiating with the manufacturer on what changes were needed. Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Case Study: The Therac-25 (5 of 6) Observations and Perspective Minor design and implementation errors usually occur in complex systems; they are to be expected The problems in the Therac-25 case were not minor and suggest irresponsibility Accidents occurred on other radiation treatment equipment without computer controls when the technicians: – Left a patient after treatment started to attend a party – Did not properly measure the radioactive drugs – Confused micro-curies and milli-curies Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Case Study: The Therac-25 (6 of 6) Discussion Question If you were a judge who had to assign responsibility in this case, how much responsibility would you assign to the programmer, the manufacturer, and the hospital or clinic using the machine? Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Increasing Reliability and Safety (1 of 8) Professional techniques Importance of good software engineering and professional responsibility User interfaces and human factors Redundancy and self-checking Testing – Include real world testing with real users Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Increasing Reliability and Safety (2 of 8) Professional techniques Management and communication High reliability organization principles – preoccupation with failure – loose structure Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Increasing Reliability and Safety (3 of 8) Safety-critical applications Identify risks and protect against them Convincing case for safety Avoid complacency Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Increasing Reliability and Safety (4 of 8) Specifications Learn the needs of the client Understand how the client will use the system Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Increasing Reliability and Safety (5 of 8) User interfaces and human factors User interfaces should: – provide clear instructions and error messages – be consistent – include appropriate checking of input to reduce major system failures caused by typos or other errors a person will likely make Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Increasing Reliability and Safety (6 of 8) User interfaces and human factors The user needs feedback to understand what the system is doing at any time. The system should behave as an experienced user expects. A workload that is too low can be dangerous. Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Increasing Reliability and Safety (7 of 8) Redundancy and self-checking Multiple computers capable of same task; if one fails, another can do the job. Voting redundancy Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Increasing Reliability and Safety (8 of 8) Testing Even small changes need thorough testing Independent verification and validation (IV & V) Beta testing Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Trust the Human or the Computer System? Traffic Collision Avoidance System (TCAS) Computers in some airplanes prevent certain pilot actions Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Law, Regulation, and Markets Criminal and civil penalties – Provide incentives to produce good systems, but shouldn’t inhibit innovation Regulation for safety-critical applications Professional licensing – Arguments for and against Taking responsibility Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Dependence, Risk, and Progress (1 of 2) Are We Too Dependent on Computers? – Computers are tools – They are not the only dependence ▪ Electricity Risk and Progress – Many new technologies were not very safe when they were first developed – We develop and improve new technologies in response to accidents and disasters – We should compare the risks of using computers with the risks of other methods and the benefits to be gained Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Dependence, Risk, and Progress (2 of 2) Discussion Questions Do you believe we are too dependent on computers? Why or why not? In what ways are we safer due to new technologies? Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved Copyright This work is protected by United States copyright laws and is provided solely for the use of instructors in teaching their courses and assessing student learning. Dissemination or sale of any part of this work (including on the World Wide Web) will destroy the integrity of the work and is not permitted. The work and materials from it should never be made available to students except by instructors using the accompanying text in their classes. All recipients of this work are expected to abide by these restrictions and to honor the intended pedagogical purposes and the needs of other instructors who rely on these materials. Copyright © 2018, 2013, 2008 Pearson Education, Inc. All Rights Reserved