Critical System Validation PDF

Critical System Validation What we mean by critical systems? It is a system that characterized by high reliability and holds this reliability as evolve without more cost. It is a system in which if it’s failure results leads to significant economic losses, physical damage, or threats...

Critical System Validation What we mean by critical systems? It is a system that characterized by high reliability and holds this reliability as evolve without more cost. It is a system in which if it’s failure results leads to significant economic losses, physical damage, or threats to human life. The most important property of critical systems is Dependability. This characteristic include: Why it is important? Two main reasons: 1. System if unreliable, unsafe, and unsecure will be not acceptable by users. 2. System failure costs will be very high. Failures can be in user (can’t operate correctly), hardware (problem in design devices), and software (errors in design, coding, specification). According to the previous reasons, we should validate the critical system carefully against reliability, safety, and security. Validation cost of the critical systems involved. Critical system types Critical systems validation perspectives Reliability validation: This validation concerns with the number of times that the system fails to deliver specified services. To ensure from that, we need a special dataset not used in the testing process in order to evaluate reliability. In reliability validation, we want to check if the Measurable reliability meets the specification or not. Also, if the measurable reliability is enough to satisfy the users need. Safety validation: System works in a method that the faults do not happen or if it is happened, the consequence will be minimized. There is a difficulty to specify and assessing safety. Security validation: System and its data should be secure against the internal and external attacks. Critical systems validation techniques These techniques applied after the system is ready. Static technique: aimed to improve the quality of code without executing it. To do that, we depend on the Reviewing process by: - Design review and check the program and documentation. - Mathematical parameters proofs may be prepared to support validation. - Finding errors in the system and identifying possible problems will happen during execution. Critical systems validation techniques Dynamic technique: Focused on observing system behavior if its meet the specification by: - Statistical testing - Runtime checking - Scenario testing To apply these testing, we need executable software Critical systems validation techniques Process validation: Minimize the probability of process errors that affects on dependability of system using a good design development processes. Safety Validation Static technique for safety validation Proving the safety of the system by the testing process is difficult, the reason is related to the mechanism of the testing process, where the testing cases of the system is determined by selection. Therefore, testing all cases in the system is very costly and may be impossible. To solve this matter, system should be reviewed statically for correctness, in order to check that the unsafe situations will not be occur. Reviewing process include:  Review for the correct intended function.  Review for understandable and maintainable structures of the system.  Review to verify algorithms and data structures design against the specification.  Review to check code consistency with algorithms and data structures design.  Review the efficiency of system testing. Guidance of the reviewing process:  Make the system simple.  Use a simple techniques for software development.  Do not think that fault-tolerance technique is always safe. Hazard analysis  To assured from the effectiveness of system safety, we depend on the Hazard analysis, which is: - The first step that is used to evaluate the system risk. - Concerned with discovering the root of the risk in the system.  Hazard analysis include three technique to reduce the system risk: - Hazard avoidance - Hazard identification and removal - Damage limitation  Hazard avoidance During the design stage, there is a possibility to find some objects that can produce a risk on a system. It should be avoided  Hazard identification and removal Detection and removing the reasons that lead to making system under the risk before it is happening  Damage limitation The system should include safety and protection features that reduce the impacts of the risks if it's happened All hazard techniques can be used together. Safety case of the system  In all computer-based system, such as transportation systems (cars, trains, aviation), medical industry, etc. safety should be taken into account.  Safety in the system required an evidence to support it, to justify that the system is safe for work in a particular environment. Example: car is safe to drive in road evidence: auto emergency braking, stability control, airbags, etc.  Safety feature should be revisited when the system updated. Gas warning system Example System to warn of poisonous gas. Consists of a sensor, a controller and an alarm Two levels of gas are hazardous Warning level - no immediate danger but take action to reduce level Evacuate level - immediate danger. Evacuate the area The controller takes air samples, computes the gas level and then decides whether or not the alarm should be activated Gas sensor control Gas_level: GL_TYPE ; loop -- Take 100 samples of air Gas_level := 0.000 ; for i in 1..100 loop Gas_level := Gas_level + Gas_sensor.Read ; end loop ; Gas_level := Gas_level / 100 ; if Gas_level > Warning and Gas_level < Danger then Alarm := Warning ; Wait_for_reset ; elsif Gas_level > Danger then Alarm := Evacuate ; Wait_for_reset ; else Alarm := off ; end if ; end loop ; Graphical argument Gas_level > Warning and Unsafe state Alarm = off or or or Path 1 Path Path 2 3 Gas_level > Gas_level > Danger Warning and Gas_level < Danger Alarm = Evacuate Alarm = off Alarm = Warning contradicti contradicti on on Condition checking Code is incorrect. Gas_level = Danger does not cause the alarm to be on Security Validation  Security validation focused on the protection of the system and its data against possible attacks. Security of the system can be validated using static or dynamic techniques. Security and Safety are common in some aspects, where both of them aimed to prevent the system from entering unsafe or unsecured states. Security validation methods  Experience based validation. Validation teams who analyze and review the systems against the possible attacks should have a high degree of experience.  Tool based validation. Systems should be validated using the available security tools In some parts. (automatic validation). Tiger teams. a tiger team is a group that tests an organization's ability to protect its assets against physical or information security. In this context, the tiger team tries to break the security of the system by simulating attacks. Safety Vs. Security Safety Security Unplanned (accident)  Planned Protect human life  Protect systems, Depend on the information application  generic Internal  External Reliability Validation 28 Dynamic technique for reliability validation Involves executing the system, in order to assess if it is reached the required level of reliability. To do that: we need a particular dataset for assessing. Statistical tests use to prove the reliability by measuring the number of errors in the system that leads to failure situations. 29 Dynamic technique for reliability validation How can we identify that the system has the reliability? For each system, we should identify a threshold value, if the number of failures in the system exceeds this value, we can say that the system is not reliable, otherwise, the system is reliable. 30  0 threshold 100 50 Reliability validation process: Identify Prepare Compute Apply test operation test observed to system al profile dataset reliability  Step1: Establish the operational profile for the system. (Identify) Set of data that will be used in the test. Dataset must match the real data used in the system. Can be obtained from data collected from another system. Reliability evaluated on different data sizes. Generated automatically if possible in order to reduce errors. 32 Step2: Construct dataset for testing that reflects the operational profile. Step3: Test the system and observe the number of failures and the times of these failures. Step4: Compute the reliability after statistically significant number of failures have been constructed. 33 Problems in reliability validation Uncertainty of operational profile Is the operational profile reflect the real system usage High cost of data testing We cannot test all cases in the system Statistical uncertainty Very difficult to obtain the number of failures that allows us to compute the reliability which is statistically significant. 34 Formal Method and Validation  The formal method used to improve the system quality through specification by reduce the errors and discover the weakness. Formal methods are expensive to use. So, not all types of systems can use it in the development process. Only the critical systems use it (based on the importance of critical systems) because of its useful and increases the confidence in the system, especially, in safety, security, and reliability. One of the most interesting applications of formal methods in the development of tools that can generate complete test cases from the specification. Specification validation and formal verification depend on mathematical analysis of the system. Validation can be done by:  Human Judgments to determine if building the right system, especially, in simple and small systems.  Formal methods, in particular, in large and complex systems because of its required huge amounts of effort and times. To do this, several tools have been developed.  Note: formal methods do not discard human judgment, but it helps in determining if the right system has been specified. Problems with formal method validation  Not understandable by the experts of the domain.  Cost of validation and verification increases as the system size and complexity increase. Safety proofs. Aimed to show that the system will not enter the unsafe situations by using the proof by contradiction strategy. To do that we will use the diagrams.