NUREG-1855 Training Course

Web-based Course on NUREG-1855 Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decisionmaking Text-Only Version Last Updated: April 12, 2017 Page 1 of 83 Title Slide Welcome to the web-based course on NUREG-1855, titled “Guidance on the Treatment of Uncertainties Ass...

Web-based Course on NUREG-1855 Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decisionmaking Text-Only Version Last Updated: April 12, 2017 Page 1 of 83 Title Slide Welcome to the web-based course on NUREG-1855, titled “Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decisionmaking.” This course is divided into 18 modules. Each module contains 1-2 main level slides which are designated with a letter A following the title number. Further details supporting the primary information on the main slides is provided in slides with letter B, C, D, etc. following the slide number. U.S. NRC Logo: United State Nuclear Regulatory Commission Protecting People and the Environment EPRI Logo: Electric Power Research Institute Page 2 of 83 Slide 1A - Need for Guidance 1 Why is this guidance needed and what is the purpose of NUREG-1855 (and the associated EPRI Reports)? This course is needed because uncertainties associated with the risk analysis need to be recognized and understood so that the decision maker is able to make a sound, riskinformed decision. A risk-informed decision relies on quantitative risk acceptance guidelines that are used to compare with the risk results from the licensee’s risk analysis, for example, a Probabilistic Risk Assessment. There are, however, uncertainties associated with the PRA model, and therefore, with the results. For example, the decision maker needs to: understand to what extent the results are impacted by the uncertainties understand whether there are uncertainties that are impacting the results that may challenge the acceptance guidelines and determine if the driver for the large uncertainties can be identified and remediated. Guidance is needed for how the applicant will address uncertainties and how the staff will integrate the treatment into their risk-informed decisionmaking process. The primary objective of NUREG-1855 and the associated EPRI reports is to provide guidance on how uncertainties associated with the PRA results are treated in the Riskinformed Decisionmaking process. NUREG-1855 and the EPRI reports can be found here: NUREG-1855, “Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decision Making,” [https://www.nrc.gov/reading-rm/doccollections/nuregs/staff/sr1855/] EPRI Report 1013492, “Probabilistic Risk Assessment Compendium of Candidate Consensus Models,” [http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010 13492/] EXIT: This link is external to the NRC website EPRI Report 1016737, “Treatment of Parameter and Model Uncertainty for Probabilistic Risk Assessments,” Page 3 of 83 [http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010 16737/] EXIT: This link is external to the NRC website EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 4 of 83 Slide 1B - Associated EPRI Reports The associated EPRI reports provide guidance on how uncertainties associated with PRA results are treated in the Risk-informed Decisionmaking process. Links to the EPRI reports: EPRI Report 1013492, “Probabilistic Risk Assessment Compendium of Candidate Consensus Models,” [http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010 13492/] EXIT: This link is external to the NRC website EPRI Report 1016737, “Treatment of Parameter and Model Uncertainty for Probabilistic Risk Assessments,” [http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010 16737/] EXIT: This link is external to the NRC website EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Link to NUREG-1855: NUREG-1855, “Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decision Making,” [https://www.nrc.gov/reading-rm/doccollections/nuregs/staff/sr1855/] Page 5 of 83 Slide 2A – Course Learning Objectives 2 What should you learn from this course? The purpose of this course to provide a high level understanding of the content in NUREG-1855 and the associated EPRI reports. After completing this course, you will be able to identify: The importance of uncertainties in the decisionmaking process Key concepts associated with risk-informed decisionmaking Types of uncertainties and their impact on PRA results and The approach used to treat PRA uncertainties in the decisionmaking process This course is not intended to provide guidance on how to: Implement the detailed guidance contained in NUREG-1855 (and the associated EPRI reports) Quantify the uncertainties in developing the PRA model Page 6 of 83 Slide 2B - More Information on Uncertainties in PRA You should also gain an appreciation for the following: Decision-making on complex issues is always subject to uncertainty. Underlying uncertainties exist whether or not it is acknowledged. The purpose of treating uncertainties in PRA is to ensure that the decisionmaker clearly understands whether the PRA uncertainties have: been adequately addressed in the application, and the potential to affect the decision. PRA complements deterministic decision criteria by providing structure to the issue of uncertainty. When appropriately addressed, the uncertainties analysis adds technical support and value to risk-informed decisionmaking. The analysis of uncertainties in risk models allows the decisionmaker to make a more robust decision. Page 7 of 83 Slide 3A - Background Information To best benefit from this course, it is useful to have a foundational knowledge of riskinformed decisionmaking. There are some key concepts that you should understand before taking this course. Please answer each of the following five questions in order to explore and review these concepts. Question #1 - PRA Models What does a PRA model do? Choose all that apply: A) Examines a spectrum of possible outcomes with a discrete set of representative scenarios. B) Ignores the impact of modeling assumptions to produce a best estimate risk outcome. C) Creates a new source of uncertainty that needs to be accounted for in a decisionmaking process. The correct answer is: A. Examines a spectrum of possible outcomes with a discrete set of representative scenarios. Fundamentally, a PRA is a model that represents the accident sequences which could occur as a result of the various events (initiating events) that could challenge the normal operation of the plant. A PRA model is based on engineering (deterministic) analyses, such as the identification of system success criteria (e.g., based on thermo-hydraulic analyses), containment capacity (e.g., based on structural analysis) and plant specific information so that the model represents the as-built and as-operated plant. The PRA has probabilistic models whose parameters (e.g., initiating event frequencies, component failure rates, etc.) are estimated based on available knowledge or data. Page 8 of 83 The PRA results provide insights on both what is important in causing the risk and what can reduce the risk, and provide insights to the magnitude (potential impact) of parameter, model, and completeness uncertainties. A full-scope PRA addresses three levels of risk (i.e., core damage, radiological releases, and consequences) from both internal and external hazards, and all operating states (at-power, low power and shutdown) Not all risk-informed activities require a full-scope PRA. The needed scope is dependent on the specific decision/application under consideration. The acceptability of the base PRA model is determined by the current state-ofpractice (as codified in the PRA Standard and as endorsed by Regulatory Guide 1.200). Link to Regulatory Guide 1.200: Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities [https://www.nrc.gov/docs/ML0904/ML090410014.pdf] Question #2 - Difference between Risk-informed and Risk-based Risk-based and Risk-informed decisionmaking methods use the same information to make a decision. A) True B) False The correct answer is: FALSE A "risk-based" approach to regulatory decisionmaking is an approach in which such decisionmaking is solely based on the numerical results of a risk assessment. Page 9 of 83 A "risk-informed" approach to regulatory decisionmaking is an approach in which decisionmaking is not solely based on the numerical results of a risk assessment. A risk-informed approach takes into account risk insights (including the numerical results) along with other factors such as defense-in-depth, monitoring, safety margins. Question #3 - Risk-informed Decisionmaking Principles What are the principles, as outlined in Regulatory Guide 1.174, in the risk-informed decisionmaking process? Choose all that apply: A) Regulation is met or explicit rule exemption B) Consistent with defense in depth C) Maintain safety margins D) Performance monitoring E) Deterministic criteria F) As low as is reasonably achievable G) Risk increases are small and consistent with safety goal H) Minimization of risk The correct answer is: There are five principles constituting the integrated decisionmaking process: Regulation is met or explicit rule exemption Consistent with defense in depth Maintain safety margins Performance monitoring Risk increases are small and consistent with safety goal Page 10 of 83 The existence of uncertainties does not stop us from using the results of a risk assessment (or a deterministic assessment) in our decisionmaking. We employ the other elements of the risk-informed decisionmaking process to address the PRA uncertainties that can impact the decision. See Regulatory Guide 1.174 for more about the risk-informed decisionmaking principles. PRA results are only one portion of a riskinformed decisionmaking process. All the principles are used in the decisionmaking process. Link to Regulatory Guide 1.174: Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities [https://www.nrc.gov/docs/ML0904/ML090410014.pdf] Question #4 - PRA Uncertainties Engineering analyses account for uncertainties. A) True B) False The correct answer is: TRUE Knowledge is imperfect regardless of how an analysis is conducted. Uncertainties exist in both engineering and risk analyses. Consequently, uncertainties from both engineering and risk analyses, need to be addressed in the decisionmaking process. However, as opposed to deterministic analyses, PRAs make the impact of uncertainties on the results explicitly observable by accounting for uncertainties in the quantification of the risk metrics. Definition of Uncertainty: Uncertainty is the understanding that knowledge is imperfect, and this imperfect knowledge needs to be considered in decisionmaking. Question #5 – Risk-informed Terminology Page 11 of 83 Which of the following statements are false? Choose all that apply: A) A frequency and probability have the same meaning. B) Uncertainties are only associated with our ability to accurately predict the likelihood of equipment performance. C) A risk analysis and a PRA are identical. The correct answer is: All the answers are false. A frequency and probability DO NOT have the same meaning. Frequency is a rate of occurrence of an event, and probability is an estimate of the likelihood that an event (for example 2 heads in 5 coin tosses) will happen at any given time. Uncertainties are NOT only associated with our ability to accurately predict the likelihood of equipment performance. There are uncertainties, for example, in our ability to predict human performance, phenomena, hazard frequency and magnitude. A risk analysis and a PRA are NOT identical. A PRA is one type of risk analysis. A risk analysis can be a qualitative or quantitative analysis of the risk metric(s). A PRA, however, is a type of quantitative analysis of the risk metric(s). Definitions for the risk terms used in this course (and used in NUREG-1855 and the associated EPRI reports), can be found in NUREG-2122 (“Glossary of Risk-Related terms in Support of Risk-Informed Decisionmaking”). This glossary is a great reference for you as a risk-informed decision maker! Link to NUREG-2122: NUREG-2122, “Glossary of Risk-Related terms in Support of Risk-Informed Decisionmaking,” [http://pbadupws.nrc.gov/docs/ML1215/ML121570620.pdf] Page 12 of 83 Slide 4A – Decisionmaker Guidance 4 What type of guidance is provided to a decision maker? This course explains the guidance that is provided for addressing PRA uncertainties as part of the risk-informed decision making process. It is important to have a basic understanding of this process. This decisionmaking process has five main elements. Page 13 of 83 Slide 4A (Continued) - Risk-informed Decisionmaking Process Let’s take a closer look at the risk-informed decisionmaking process. First, the decision under consideration is defined. Secondly, the regulatory requirements that can be impacted by the decision are identified, including assessing how the ability to meet the requirement may be impacted. Third, a risk analysis is performed to evaluate how the decision may impact the risk to public health and safety from the nuclear power plant. This analysis includes assessing the PRA uncertainties in two different ways, both from a deterministic perspective and from a probabilistic perspective. These perspectives require assessing the decisions impact on defense in depth and safety margins. Both perspectives require an assessment of uncertainties in their own way. This course is focused on the probabilistic perspective. Fourth, identify how monitoring the performance of the impacted equipment or operation would be performed. The assessment of uncertainties be incorporated into the implementation and monitoring program and that program should be accounted for in the assessment of uncertainties. Fifth, the decision maker considers the PRA uncertainties along with the other elements as part of the risk-informed decisionmaking process, which includes an assessment of the uncertainties. Page 14 of 83 Slide 5A – Uncertainties Treated Differently 5 Uncertainties exist in both deterministic and probabilistic analyses, are they treated differently? Uncertainties exist in both deterministic and probabilistic analyses. However, each of these approaches addresses uncertainties differently. Deterministic decisionmaking addresses the uncertainties using calculations that are deliberately conservative to establish absolute criteria. In this approach, the possibility that structures, systems and components may operate past their regulatory limit is generally not considered by the decisionmaker. Probabilistic decisionmaking addresses the uncertainties using calculations that are intended to be realistic to establish a range of possible outcomes that are factored into the decisionmaking process. In this approach, the possibility that structures, systems and components may operate past their regulatory limit is generally considered by the decision maker. Page 15 of 83 Slide 5B – Deterministic Approach Deterministic Approach The deterministic approach addresses uncertainties by generally establishing conservative bounds (e.g., safety margins) to compensate for the uncertainties. For example, a battery is considered depleted once the established regulatory limit has been exceeded (although the limit is considered to be conservative with a certain amount of margin). Page 16 of 83 Slide 5C – Probabilistic Approach Probabilistic Approach The probabilistic approach addresses uncertainties by establishing a range of possible outcomes with associated probabilities for each of the various outcome. In this example, the battery is considered depleted at different limits where there is a level of confidence established for each limit. For an example where a power from a battery is needed to ensure system functionality, at limit “x,” there is 5% chance of depletion, at limit “y,” there is an average chance of depletion, and at limit “z,” there is a 95% chance of depletion. A decisionmaker will likely be more comfortable at the “x” limit than the “z” limit, but there is not hard transition point between acceptability and unacceptability. Page 17 of 83 Slide 6A – Types of Uncertainties 6 There are different types of uncertainties associated with PRAs: What are they? Fundamentally there are three types of uncertainties associated with PRAs that reflect our imperfect knowledge. Completeness uncertainties: Addresses the confidence in our knowledge (or ability) that all the risk significant events and physical phenomenon have been identified and included in the risk analysis. Parameter uncertainties: Addresses the confidence in our state of knowledge (or ability) to accurately predict the frequency of an event, the probability of equipment successfully performing, or the probability of a successful human action. Model uncertainties: Addresses the confidence in our state of knowledge (or ability) to accurately model the performance of structures, systems or components under adverse conditions. given our lack of understanding of the conditions (phenomenon) being modeled or the nature of the possible mechanisms causing the structures’, systems’ and components’ failure. EPRI Report 1016737 provides a list of uncertainties for internal events EPRI Report 1026511 provides a list of uncertainties for fire, seismic, low-power shutdown and level 2. Link to EPRI Report 1016737: EPRI Report 1016737, “Treatment of Parameter and Model Uncertainty for Probabilistic Risk Assessments,” [http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010 16737/] EXIT: This link is external to the NRC website Link to EPRI Report 1026511: EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 18 of 83 Slide 6B – Completeness Uncertainties Example Completeness uncertainties are those associated with phenomena or failure modes that are not included in the PRA model. Some may not be included because there are no analytical methods for dealing with them. Others are simply because we don’t know about them; they are the “unknown unknowns”. For example, modeling of degradation mechanisms of a material is based on an understanding of what the underlying environmental conditions are for that material (e.g., temperature, chemical environment, etc.). If the possible range of those conditions are not completely identified, then important degradation mechanisms can be missed. The Davis-Besse plant found severe degradation of the reactor vessel head as a result of prolonged exposure to concentrated boric acid. Operating experience reveals instances in which corrosion rates were significantly underestimated for identified boric acid leaks because of erroneous assumptions regarding the nature of the leakage, environmental conditions, the relationship between the actual leakage and experimental data, or other factors. As a consequence, in some instances, carbon steel components have been corroded to a much greater extent than anticipated. Page 19 of 83 Slide 6C – Parameter Uncertainties Example The various parameters used to quantify the PRA model, including the initiating event frequencies, component failure probabilities, and human error probabilities cannot be known with certainty. This uncertainty is characterized by a probability distribution on the parameter value. For an example from EPRI 1026511, the frequency of a large earthquake is a parameter that can have substantial uncertainty around it. Generally, very large earthquakes are rare events, so there is not sufficient data to produce a statistically robust frequency estimate. Because we do not know the true frequency, we assign an uncertainty distribution to capture the level of confidence in that estimate. Page 20 of 83 Slide 6D – Model Uncertainties Example Model uncertainties result from a lack of knowledge of how structures, systems and components behave under the conditions arising during the development of an accident. Model uncertainties can arise because the phenomenon being modeled itself is not completely understood. For some phenomena, some data or other information may exist, but it needs to be interpreted to infer behavior under conditions different from those in which the data were collected. Model uncertainties may also arise because the nature of the failure mode is not completely understood or is unknown. An example of model uncertainty, taken from EPRI 1026511, is the different models used to predict core melt progression. MELCOR and MAAP are two models used to determine the core melt progression for severe accidents. These models differ in some fundamental assumptions in how the underlying phenomena are treated (e.g., to what extent can solid debris block a flowpath, how particulate debris move, how crust formation effects movement of corium through the reactor vessel, etc.) due to limits in our basic understanding of the physics and computational limitations. While both codes are benchmarked against available data, and both perform well on those cases, there is limited data that might resolve these differences. These underlying differences in capturing the relevant phenomena can translate to differences in the output (e.g., temperatures, amount of hydrogen generated, etc.) that is then used in the PRA. Page 21 of 83 Slide 7A - Overall Guidance in NUREG-1855 7 What is the overall process in NUREG-1855 and the associated EPRI reports? The process for treating uncertainties in risk-informed decisionmaking is comprised of seven stages divided into three parts. The first part provides guidance to both the licensee and the NRC staff and is only comprised of Stage A which addresses whether the guidance in NUREG-1855 and the associated EPRI reports should be used for the risk-informed decision under consideration. The second part provides guidance to the licensee and is comprised of Stage B through Stage F and represent the licensee’s process. In Stage B, guidance is provided for the licensee to assess whether the scope and level of detail of the PRA is adequate to support the application. In Stage C, guidance is provided to the licensee to assess the significance of the missing scope and missing level of detail. In Stages D and E, guidance is provided to the licensee on how to treat parameter and model uncertainties. Stages C through E are iterative. The results of stages C D and E are feed into Stage F, where guidance is provided to the licensee on the possible strategies for addressing the key uncertainties. Stage G provides a discussion of the approach used by the staff in their risk review of the licensee’s application referred to her as the NRC Decisionmaking process. Stage G focuses on the determination of: PRA Acceptability Appropriate Use of Bounding Analyses of Inadequate Scope Treatment of Parameter and Key Model Uncertainties The NRC Decisionmaking process iterates with the Licensee process until a decision is reached. The guidance in Stages A through G are based on the NRC risk-informed decisionmaking process, and the treatment of uncertainties is consistent with the ASME/ANS PRA standard. Page 22 of 83 Slide 8A - Relation of NUREG-1855 and Associated EPRI Reports 8 How are the NUREG-1855 and associated EPRI reports related? NUREG-1855 and the associated EPRI reports are designed to be used together in addressing uncertainties in a risk-informed application. They facilitate: Understanding the Application The role of the PRA Standard The role of the PRA Model NUREG-1855 provides the overall guidance on the process and focuses on Stage A through G, while the associated EPRI reports provide detailed guidance on only stages B through F of the process. The key contributions from the NUREG and EPRI reports are: NUREG Report: Uncertainty from Non-Modeled Risk Contributors Characterization and Propagation of Parameters Identification of Key Model Uncertainties EPRI Reports: Detailed and Approximate Methods for Parameters Identification and Characterization of the Model Uncertainties Detailed Example Page 23 of 83 Slide 8B - More Information on Uncertainties in PRA EPRI 1026511 Steps in Relation to Stages of NUREG-1855, Rev. 1 The information below provides the process in EPRI 1026511 that a licensee should follow in preparing a risk-informed application, and how uncertainty is considered in each step of the process. The diagram also provides links between the licensee process and the steps in NUREG-1855. Step 1 (NUREG-1855 Step B1): Define applications: -Acceptance guidelines -Scope of risk contributions -Cause-effect relationship Step 2 (NUREG-1855 Step B2): Assess adequacy of existing PRA to model cause-effect relationship If the cause and effect relationships in the PRA are not adequate, refine model or redefine application (NUREG-1855 Step B3) and go back to step 2. If the cause and effect relationships in the PRA model it are adequate, the licensee needs to asses if they need a full scope PRA or if the existing PRA scope is adequate? If the existing PRA scope is not adequate, then proceed to step 3. Step 3: Perform initial comparison with acceptance guidelines. Identify significant contributors and role of affected function(s). Once step 3 is finished, conduct step 4. Step 4 (NUREG-1855 Step C): Assess adequacy of existing PRA to model cause-effect relationship. If the cause and effect relationships in the PRA are not adequate, refine model or redefine application (NUREG-1855 Step B3) and go back to step 2. Page 24 of 83 If the cause and effect relationships in the PRA are adequate, proceed to step 5. If the answer to the previously asked question, is the existing PRA scope adequate, was yes, then insights from EPRI 1016737 are incorporated and the licensee would move directly to step 5 from that question. Step 5 (NUREG-1855 Steps D and E): Comparison with acceptance guidelines and identify significant contributors -Identify conservatisms -Parameter uncertainty -Model uncertainty If the Acceptance Guidelines are exceeded (Section 5), then the licensee needs to refine the model or redefine the application and return to Step 5. If the acceptance guidelines are not exceeded, proceed to Step 6. Step 6 (NUREG-1855 Steps F): Integrated decision-making Link to EPRI Report 1026511: EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 25 of 83 Slide 9A - Activities that Use NUREG-1855 9 When is the guidance in NUREG-1855 and associated EPRI reports used? Stage A provides guidance to both the licensees and the NRC staff for determining whether the detailed guidance in NUREG-1855 and the associated EPRI reports should be used for the risk-informed application under consideration. The goal of this stage is to determine whether the results from a PRA are used in the risk-informed application and to determine how the PRA results are to be used to support the decision. Page 26 of 83 Slide 9B - Stage A Process Stage A involves two steps: Step A-1 determines the type of risk metric results used in the application. If the risk metric is independent of the PRA, treatment of uncertainties is addressed through a different process than NUREG-1855 (See Section 3.4). If the risk metric is PRA dependent, proceed to step A-2. Step A-2 determines how the PRA results are used to support the decision requested in the application. If the PRA results are continuous or immediately evaluated, treatment of uncertainties is addressed through a different process than NUREG-1855 (See Section 3.4). If the application of the PRA results is a regulatory submittal, proceed to step B, assessing the PRA Scope and Level of Detail. Although not all decisions explicitly use the guidance in NUREG-1855 and the associated EPRI reports, the guidance in these documents can be used in a generic fashion as described in the next screen. For those decisions that will use the guidance in NUREG-1855 and the associated EPRI reports, the analyst then follows Stages B through F. Page 27 of 83 Slide 9B – Step A-1: Type of Results In this step, the analyst determines if the risk results used in the application are dependent on development of a PRA, either by evaluating changes to the core damage frequency and/or large early release frequency values or by utilizing importance measures to gain insights into risk significant components or actions. Some example risk-informed activities include: Component Risk Ranking (e.g., motor- operated valves) -- Utilizes the FussellVesely importance measure (IM) calculated from a PRA to rank components. Regulatory Guide 1.174 (licensing basis changes) -- Uses core damage frequency and large early release frequency calculated from the baseline PRA and the potential change as a result of the proposed modifications to the plant design or operation reflected in the PRA. 10 CFR 50.69 (special treatment of structures, systems and components) -- Uses Fussell-Vesely and risk achievement worth importance measures calculated from a PRA. Also uses the core damage frequency and large early release frequency calculated from the baseline PRA and the potential change as a result of the proposed modifications to the plant design or operation reflected in the PRA. Notice of Enforcement Discretion (NOED) – Uses an incremental conditional core damage probability. Page 28 of 83 Slide 9C – Step A-2: How Results Are Used In this step, the analyst determines how the application, e.g., decision, uses the PRA results based on the following criteria: 1. The risk metrics from the PRA are continuously being evaluated such that at any time, the risk is known – not subject to the detailed approach discussed in NUREG1855. Example answers for this criteria for various activities would be: Component risk ranking, no; Reg Guide 1.174, no; 10 CFR 50.69, no; NOED, no. 2. The decision under consideration is based on reviewing the PRA risk results against specified regulatory acceptance guidelines – subject to the detailed approach discussed in NUREG-1855. Example answers for this criteria for various activities would be: Component risk ranking, only importance measures; Reg Guide 1.174, yes; 10 CFR 50.69, yes; NOED, possibly. 3. The risk significance of a decision is being evaluated at the time of occurrence of an event and there is insufficient time to evaluate the impact of uncertainties – not subject to the detailed approach discussed in NUREG-1855. Example answers for this criteria for various activities would be: Component risk ranking, no; Reg Guide 1.174, no; 10 CFR 50.69, no; NOED, yes. Page 29 of 83 Slide 9D – Generic Use Stage A starts with a determination of the type of risk-informed activity that is being evaluated (non-risk-informed activities are outside the scope of this report) and the type of risk metric results used in the application. If the risk metric results do not depend on the development of a PRA, the detailed approach in NUREG-1855 does not apply. Next, with regard to how the PRA is being used in the decision, if the activity involves a continuous evaluation of risk or an evaluation of risk at the time of an event, the detailed approach in NUREG-1855 does not apply. However, the detailed approach in NUREG1855 can be generically applied for all risk-informed activities. Page 30 of 83 Slide 10A – Generic Application of NUREG-1855 10 Is the guidance in NUREG-1855 (and associated EPRI Risk-informed activities not covered under Stage A still require an assessment of the uncertainties, but, perhaps to a lesser degree. For these activities, the guidance in NUREG-1855 is not applied verbatim; the guidance is generic and can be broadly applied. This generic process is comprised of the following three steps: 1. Understanding the risk-informed activity. 2. Understanding the sources of uncertainties. 3. Addressing the uncertainties in the decisionmaking. Page 31 of 83 Slide 10B – Understanding the Risk-Informed Activity Need to identify what results are needed to evaluate the specific aspect of plant design or operation being assessed and how they are to be used to inform the decision. There can be occasions where qualitative risk insights are sufficient and thus a qualitative risk method could be sufficient to make a risk-informed decision. The level of detail of the risk assessment can vary because of choices in the modeling assumptions and approximations made in order to limit the need for potentially resource-intensive detailed analysis. Page 32 of 83 Slide 10C – Understanding Sources of Uncertainties The effect of parameter, model and completeness uncertainties on the risk assessment results must be identified to make proper risk-informed decisions. Identification and evaluation of these uncertainties for risk assessments other than PRA follows the same approach as provided in Stages C, D and E. It is necessary to assess whether the uncertainties have the possibility of changing the evaluation of risk significantly enough to alter a decision. Page 33 of 83 Slide 10D – Addressing Uncertainties It is important to understand how uncertainties can affect the decision – the proximity of the mean value to a quantitative acceptance guideline can influence the depth of analysis required to make a decision. The missing scope or level-of-detail items must be addressed in the decision process. Screening assessments can be used to show that the missing items are not important to the decision. In some cases, the risk-informed activity can be altered such that the missing scope items do not affect the decision process. The results of sensitivity studies performed to address model uncertainties can confirm that the acceptance guidelines are still met even when alternative assumptions are used (i.e., the conclusion generally remains the same with respect to the acceptance guideline). Page 34 of 83 Slide 11A – PRA Scope and Level of Detail 11 Does the missing PRA scope and level of detail matter for the application? Stage B provides guidance to the licensee to determine whether the PRA scope and level of detail is sufficient to support the risk-informed decision under consideration. The goal of this stage is to determine whether the hazard groups, initiating events, component failure modes, accident sequences, human reliability analysis, etc. are sufficient to support the risk-informed decision under consideration. Page 35 of 83 Slide 11A (Continued) Stage B Process Stage B, applicability of approach, consists of three steps: Step B-1 involves understanding the risk-informed application and decision. Step B-2 involves identifying the PRA scope and level of detail needed for the riskinformed application. Step B-3 involves assessing the PRA to determine if there is scope or level of detail needed for the risk-informed application that is missing from the PRA. If the PRA structure is satisfactory for the PRA application, continue to stage D, assessing Parameter Uncertainty. If the PRA structure is unsatisfactory for the PRA application, the licensee can redefine the application or conduct a screening analysis as defined in Stage C. Once completed, both these approaches will return the licensee to Step B-1. The licensee can also refine the PRA and proceed to Stage D, assessing Parameter Uncertainty. Associated EPRI report 1026511 provides additional guidance to the analyst on how to conduct this step. Step 1 (NUREG-1855 Step B1): Define applications: -Acceptance guidelines -Scope of risk contributions -Cause-effect relationship Step 2 (NUREG-1855 Step B2): Assess adequacy of existing PRA to model cause-effect relationship If the cause and effect relationships in the PRA are not adequate, refine model or redefine application (NUREG-1855 Step B3) and go back to step 2. If the cause and effect relationships in the PRA model it are adequate, the licensee needs to asses if they need a full scope PRA or if the existing PRA scope is adequate? Link to EPRI Report 1026511: Page 36 of 83 EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 37 of 83 Slide 11B - Step B-1: Understanding Application and Decision In this step, the analyst need to identify what structures, systems, and components and activities would be affected by a proposed application. In addition, a key aspect in this process is identifying the risk metrics needed to support the application. Stage B Step B-1: Understanding the risk-informed application and decision Step B-2: Identify the PRA scope and level of detail needed for the risk-informed application Step B-3: Does the PRA contain the scope and level of detail needed to evaluate the risk change associated with the risk-informed application? Page 38 of 83 Slide 11C – Step B-2: Identify Scope and Level of Detail In this step, the analyst needs to determine the cause-and-effect relationship between the application and the structures, systems and components and plant activities affected by the proposed risk-informed activity. This determination will then dictate the required scope of the PRA. It is then necessary to identify if the existing PRA contains the elements that would be affected by an application and are important to evaluating the risk measure(s) applicable to the decision. Stage B Step B-1: Understanding the risk-informed application and decision Step B-2: Identify the PRA scope and level of detail needed for the risk-informed application Step B-3: Does the PRA contain the scope and level of detail needed to evaluate the risk change associated with the risk-informed application? Page 39 of 83 Slide 11D – Step B-3: Addressing Missing Scope and Level of Detail In this step, the analyst, depending on whether the PRA does not contain the scope and level of detail needed to evaluate the risk change associated with a risk-informed application, chooses one of the following options: (1) upgrade the PRA to include the required scope and level of detail, (2) redefine the application such that the missing PRA scope or level of detail is not needed in the evaluation process, or (3) perform a conservative/bounding assessment of the missing items to determine if they are significant to the decision. Stage B Step B-1: Understanding the risk-informed application and decision Step B-2: Identify the PRA scope and level of detail needed for the risk-informed application Step B-3: Does the PRA contain the scope and level of detail needed to evaluate the risk change associated with the risk-informed application? Page 40 of 83 Slide 12A - Completeness Uncertainties 12 How are completeness uncertainties addressed? Stage C provides guidance to the licensee on how to address completeness uncertainties; that is, how to address the scope and level of detail items that are not modeled in the PRA. The goal of this stage is to determine whether the missing scope and level of detail are significant to the decision under consideration. Page 41 of 83 Slide 12A (Continued) - Stage C Process If Stage B determines that screening is required, the licensee will enter Stage C. Stage C involves two major steps: In Step C-1, the Licensee Performs screening analyses to determine the significance of the missing PRA scope or level of detail to the risk-informed decision. If the missing scope or level of detail are determined to be nonsignificant contributors to the decision, the licensee can move to Stage D, assessing parameter uncertainty. In Step C-2, the Licensee determines if the PRA model needs to be updated or if the application needs to be modified to address the missing PRA scope or level of detail significant to the decision. If there is not an NRC-endorsed PRA standard for the missing scope and level-of-detail, the licensee can proceed to Stage D. If a missing scope or level-of-detail item cannot be screened, the licensee will need to redefine the application by returning to Stage B. The licensee can also conduct upgrades per the PRA standard as endorsed in Regulatory Guide. 1.200, or submit the results of a conservative analysis, bounding analysis, or both to address the missing items. This upgrade process would allow the application to proceed to Stage D. Associated EPRI report 1026511 provides additional guidance to the analyst on how to conduct this step. Step 3: Perform initial comparison with acceptance guidelines. Identify significant contributors and role of affected function(s). Once step 3 is finished, conduct step 4. Step 4 (NUREG-1855 Step C): Assess adequacy of existing PRA to model cause-effect relationship. Link to Regulatory Guide 1.200 Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities [https://www.nrc.gov/docs/ML0904/ML090410014.pdf] Link to EPRI Report 1026511 Page 42 of 83 EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 43 of 83 Slide 12B – Step C-1: Perform Screening In this step, the analyst determines the risk significance of a missing scope or level of detail PRA item A screening analysis is performed to demonstrate that the particular non-modeled PRA scope or level-of-detail item can be eliminated from further consideration in the risk-informed decision. Screening analyses are either qualitative or quantitative in nature or a combination of both types. If a missing scope or level-of-detail PRA item is not screened out after it is analyzed using a bounding, conservative, or limited realistic analysis, AND it is not significant to the risk-informed decision, then the results of that analysis can be incorporated into the application as an acceptable expansion of the PRA scope or level-of-detail. Stage C Step C-1: Perform screening analyses for missing scope and level of detail items to assess significance Step C-2: Determine strategy for addressing risk from each significant un-modeled scope or level-of-detail item Page 44 of 83 Slide 12C – Step C-2: Determine Strategy In this step, the analyst addresses the risk from each significant un-modeled scope or level-of-detail item using a PRA model that is developed in accordance with a consensus standard for that item that has been endorsed by the NRC staff. If there is no PRA standard that addresses the missing scope or level-of-detail item in question, the licensee can submit the results of the quantitative screening analysis as part of the input into the decisionmaking process. When this situation occurs, the analyst can then proceed to evaluate the parameter and model uncertainties in the context of the application. Stage C Step C-1: Perform screening analyses for missing scope and level of detail items to assess significance Step C-2: Determine strategy for addressing risk from each significant un-modeled scope or level-of-detail item Page 45 of 83 Slide 13A - Parameter Uncertainties Stage D provides guidance to the licensee on how to address the parameter uncertainties associated with the PRA that is used in support of risk-informed applications. The goal of this stage is to calculate the PRA risk metrics determine how the results compare with the quantitative acceptance criteria and determine if the uncertainty of the PRA risk results, due to the underlying parameter uncertainties, impacts this comparison. Page 46 of 83 Slide 13A (Continued) – Stage D Process Once completeness has been assessed in Stage C, the licensee will enter Stage D. Stage D involves three major steps: Step D-1 characterizes the uncertainty in the quantification of the parameters in the basic events and other inputs of the PRA model. Step D-2 quantifies the risk metrics, accounting for parameter uncertainty and the State of Knowledge Correlation (SOKC). Associated EPRI report 1016737 provides detailed guidance on the state-of-knowledgecorrelation. Step D-3 compares the application risk metric results with the application acceptance guidelines. The ultimate purpose here is to determine the extent to which the mean value of the risk metric results may challenge the quantitative acceptance guidelines. This determination includes an assessment of the significance of the state-of-knowledge correlation. However, parameter uncertainty is only one type of uncertainty in the PRA results that need to be considered. Therefore, the next step in the process is consideration of model uncertainties. Associated EPRI report 1026511 provides additional guidance to the analyst on how to conduct this step. Link to EPRI Report 1026511 EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Link to EPRI Report 1016737 EPRI Report 1016737, “Treatment of Parameter and Model Uncertainty for Probabilistic Risk Assessments,” [http://www.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=0000000000010 16737/] EXIT: This link is external to the NRC website Page 47 of 83 Slide 13B – Step D-1: Characterize Parameter Uncertainties In this step, the analyst characterizes the parameter uncertainties in the form of an intervals (i.e., a range of probability values within which the actual input value lies). However, such a characterization will not lend itself to propagation of the parameter uncertainties through the PRA. Therefore, it is more typical (and necessary if uncertainty propagation is to be achieved) to characterize the uncertainties in terms of a probability distribution on the value of the quantity of concern. Stage D Step D-1: Characterize parameter uncertainty in the quantification of basic events and other PRA inputs. Step D-2: Quantify the risk metrics accounting for parameter uncertainty and the state-ofknowledge correlation. Step D-3: Compare the risk results with the acceptance guidelines. Page 48 of 83 Slide 13C – Step D-2: Quantify the Risk Metrics In this step, the analyst quantifies the risk metrics, i.e., the output of the PRA, as well as assessing the uncertainties associated with the quantification. The uncertainties of the risk metric results is a consequence of the parameter uncertainty in the PRA input values. For the simplest approach, the risk metric can be quantified as a point estimate, in which case the uncertainty associated with the risk metric can only be expressed as an uncertainty interval. For most applications ultimately requiring an NRC decision, the recommended approach is to propagate the parameter uncertainties and calculate the risk metrics in order to obtain a probability distribution from which a mean value can be derived. The effect of the state-of-knowledge-correlation on the results needs to be included in the calculation or, if neglected, justified as being negligible. Stage D Step D-1: Characterize parameter uncertainty in the quantification of basic events and other PRA inputs. Step D-2: Quantify the risk metrics accounting for parameter uncertainty and the state-ofknowledge correlation. Step D-3: Compare the risk results with the acceptance guidelines. Page 49 of 83 Slide 13D – Step D-2: Compare to Acceptance Guidelines In this step, the analyst compares the relevant risk metric results with the acceptance guidelines to be used for the particular application being considered. This comparison reveals if the acceptance guidelines are satisfied using the mean value of the risk metric results, arising from the propagation of the uncertainties in parameter values of the PRA inputs. Stage D Step D-1: Characterize parameter uncertainty in the quantification of basic events and other PRA inputs. Step D-2: Quantify the risk metrics accounting for parameter uncertainty and the state-ofknowledge correlation. Step D-3: Compare the risk results with the acceptance guidelines. Page 50 of 83 Slide 14A – Model Uncertainties 14 How are model uncertainties treated? Stage E provides guidance to the licensee for addressing sources of model uncertainties and associated assumptions related to the base PRA and the application PRA. The goal of this stage is to determine whether and the degree to which the risk metric results challenge or exceed the quantitative acceptance guidelines Any such source of model uncertainty that could cause the risk metric results to challenge or exceed the acceptance guidelines are considered to be key. Page 51 of 83 Slide 14A (Continued) Stage E Process Once parameter uncertainty has been assessed in Stage D, the licensee will enter Stage E. Stage E involves two major steps: Step E 1 identifies any potential model uncertainties as a result in constructing the base PRA model. Step E 2 identifies sources of model uncertainties key to the application. Associated EPRI reports 1016737 and 1026511 provide guidance on identification and sources of model uncertainties and a generic list of sources of model uncertainties for internal events, internal floods, internal fires, seismic, low power and shutdown and Level 2; additionally, associated EPRI report 1016737 provides guidance on performing sensitivity studies, including logical groupings of model uncertainties. At this stage of the process, the applicant has identified the sources of uncertainties and has determined their impact on the results, thereby, the impact regarding the extent to which the acceptance guidelines are met. The applicant then develops a strategy to address any key uncertainties. Once the existing PRA scope is determined to be adequate, then insights from EPRI 1016737 are incorporated and the licensee would move directly to step 5 from that question. Referring back to the flow chart in EPRI 1026511, the following steps correspond to Stage E. Step 5 (NUREG-1855 Steps D and E): Comparison with acceptance guidelines and identify significant contributors -Identify conservatisms -Parameter uncertainty -Model uncertainty If the Acceptance Guidelines are exceeded (Section 5)? Then the licensee needs to refine the model or redefine the application and return to Step 5. Page 52 of 83 If the acceptance guidelines are not exceeded, proceed to Step 6. Step 6 (NUREG-1855 Steps F): Integrated decision-making Link to EPRI Report 1026511 EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 53 of 83 Slide 14B - Step E-1: Identify Potential Model Uncertainties In this step, the analyst identifies the sources of model uncertainties in the base PRA. The base PRA is reviewed to identify and characterize the sources of model uncertainties. Some sources may be generic, and some may be plant-specific. These sources of model uncertainties are those that result from developing the PRA. The sources of model uncertainties associated with the base PRA are reviewed to identify those that are relevant to the application under consideration; that is, some sources of model uncertainties may impact part of the PRA model that is not relevant to the application. New sources of model uncertainties that may be introduced by the application are also identified. This identification is based on an understanding of the type of application and the associated acceptance guidelines. Stage E Step E-1: Identify potential model uncertainties and related assumptions and determine their significance. Step E-2: Identify model uncertainties or related assumptions that are key to the decision. Page 54 of 83 Slide 14C – Step E-2: Identify Key Uncertainties In this step, the analyst reviews the sources of model uncertainties that are relevant to the application to identify those that are key to the application. For the situation where the risk metric results from Stage D already challenges or exceeds the acceptance guidelines, it still is necessary to determine the potential significance of these sources of model uncertainties and related assumptions when they result in a further challenge to or exceedance of the acceptance guidelines. This review involves performing a quantitative analysis to identify the importance of each relevant source of model uncertainty. The relevant sources of model uncertainties that have the potential to challenge or exceed the acceptance guidelines are considered key for the application. Stage E Step E-1: Identify potential model uncertainties and related assumptions and determine their significance. Step E-2: Identify model uncertainties or related assumptions that are key to the decision. Page 55 of 83 Slide 15A Risk-informed Application Strategies 15 What strategy should licensees use in developing their risk-informed application? Stage F provides guidance to the licensee on the process of developing a risk-informed application submittal. The goal of this stage is to help ensure that adequate justification is provided for the acceptability of the risk-informed application. Page 56 of 83 Slide 15A (Continued) Stage F Process The Application Development Process, Stage F, involves three options: The first option consists of redefining the application. The second option involves refining the PRA, while the third option uses compensatory measures or performance monitoring requirements. The three options may be used separately or together, and to varying degrees, when assessing the effect of uncertainties for a particular application. The licensee may decide sufficient justification has been provided, and therefore, not pursue any of the three options. Consequently, this stage is performed in an iterative fashion. The licensee must ensure that: (1) adequate justification has been provided for the acceptability of the risk-informed application and (2) that the documentation provides a clear and concise presentation of this argument. Associated EPRI report 1026511 provides additional guidance on how a licensee can provide integrated information to the decisionmaker and how feedback from Stage F can be incorporated into Stage B. Stage F and Stages C, D, and E will impact each other and must be coherent before moving onto Stage G, the NRC’s risk informed review process. Link to EPRI Report 1026511: EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 57 of 83 Slide 15B – Option 1: Redefine the Application Redefine the Application In this option, the licensee elects to redefine the application, when a PRA model is incomplete in its coverage of significant risk contributors, the scope of a risk-informed application can be restricted to those areas supported by the PRA. The licensee must return to Stage B—as indicated by the dashed gray line—to determine scope of the redefined application and subsequently reassess the effect of all uncertainties via Stages C, D, and E. Stage F - Application Development Process Option 1: Redefine Application Option 2: Redefine PRA Option 3: Compensatory Measures or Performance Monitoring Page 58 of 83 Slide 15C – Option 2: Redefine the PRA In this option, the licensee elects to refine (i.e., upgrade) the PRA, and will either 1. address the missing scope and level of detail impacted by the application, or 2. upgrade the model of concern impacted by the application to reduce its uncertainty. The licensee must return to Stage D and then Stage E to recalculate the risk metrics, reassess the effect of model and parameter uncertainties, and determine the acceptability of the new risk measure results relative to the acceptance guidelines. Stage F - Application Development Process Option 1: Redefine Application Option 2: Redefine PRA Option 3: Compensatory Measures or Performance Monitoring Page 59 of 83 Slide 15D – Option 3: Use Compensatory or Performance Monitoring In this option, the licensee elects other means for justifying their application. Additional justification can involve, for example, compensatory measures and performance monitoring requirements for a given application. Compensatory measures can be used to neutralize the expected negative effect of some feature of plant design or operation on risk. Performance monitoring can be used to demonstrate that, following a change to the design of the plant or operational practices, there has been no degradation in specified aspects of plant performance that are expected to be affected by the change. This monitoring is an effective strategy when no predictive model has been developed for plant performance in response to a change. Stage F - Application Development Process Option 1: Redefine Application Option 2: Redefine PRA Option 3: Compensatory Measures or Performance Monitoring Page 60 of 83 Slide 15E - How a licensee can provide integrated information to the decisionmaker Associated EPRI report (1026511) provides additional guidance on how a licensee can provide integrated information to the decisionmaker. Special topics addressed include: Development of compensatory measures and performance monitoring using insights from the uncertainties analysis Assessment of defense-in-depth Dealing with very large uncertainties Appendix A provides a complete example application Link EPRI Report 1026511: EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 61 of 83 Slide 15F - Dealing with Large Uncertainties -For some contributors, core damage frequency and large early release frequency) can be very large. The causes of large uncertainties are generally associated with: There is little or no data to support its characterization although the potential for the challenge to the plant to occur is known (note rewrite of script) The need to extrapolate or interpolate from limited data for the particular use, or The need to depend on models of underlying phenomena for which there is no direct validation. -Large uncertainties can be manifested in PRA models and results in a variety of ways and can appear as uncertainties on parameters or in the model structure: Characterization of the hazard Impact of the hazard Response to hazard -The uncertainties may be so large that the analyst does not have confidence that the mean value resulting from the analysis is meaningful -Defense-in-depth is recognized as a way of addressing uncertainties in both the traditional and risk-informed approaches -The qualitative insights obtained from the understanding of the conditional core damage probability analyses provides an understanding of the margins built into the plant to enable a focus on those aspects for which defense in depth is weakest. Page 62 of 83 Slide 16A - NRC Staff Review Process 16 What process does the NRC staff use in their risk-informed review? Stage G describes the process used by the staff for determining whether a licensee’s risk-informed application demonstrates an acceptable treatment of uncertainties and whether the proposed application represents an acceptable risk to the plant. The risk-informed review process described here only involves those aspects of the staff’s risk-informed review process that specifically relate to the treatment of uncertainties. In determining whether the acceptance guidelines have been met, the staff seeks to answer the following general questions: 1. Is the scope and level of detail of the PRA appropriate for the application? 2. Is the PRA model acceptable? 3. What is the proximity of the risk metric results to the acceptance guidelines? 4. How do parameter uncertainties affect the risk metric results? 5. How do model uncertainties affect the risk metric results? 6. Is the acceptability of the application adequately justified? Note: The risk-informed review process described here only involves those aspects of the staff’s risk-informed review process that specifically relate to the treatment of uncertainties. Guidance on the other aspects of the staff’s risk-informed review process may be found in other application-specific guidance documents. Page 63 of 83 Slide 16B - Appropriateness of Scope and Level of Detail The staff addresses the acceptability of the scope and level of detail by comparing the scope and level of detail required (as requested in the specific risk-informed activity staff guidance) to that in the licensee’s submitted application PRA. This comparison identifies those PRA scope and level-of-detail items that have either not been modeled in the PRA or have been intentionally excluded for that application. The staff reviews the screening analyses provided by the licensee to determine if there is adequate justification for the exclusion of PRA items that are missing from the application. Page 64 of 83 Slide 16C - PRA Acceptability The acceptability of the base PRA is addressed in Regulatory Guide 1.200. If an unacceptability issue is identified, the staff then determines whether the issue is significant. If the staff determines that the issue is significant, the application may be rejected as a risk-informed application. If the issue is determined not to be significant, the staff continues with the risk-informed review process. Page 65 of 83 Slide 16D - The Proximity of the Risk Results to the Acceptance Guidelines The staff first determines the numerical difference between the licensee’s PRA model risk metric results and the acceptance guideline, then subsequently determines if the proximity to the guidelines is significant (i.e., challenging or exceeding the guidelines). If the proximity is not significant, the staff continues on with the risk-informed review process. If the proximity is significant, the staff determines whether adequate justification has been provided for the acceptability of its proximity. The adequacy of the justification for the proximity may not be fully evident until the staff has reviewed the licensee’s treatment of both parameter and model uncertainties. Page 66 of 83 Slide 16E - Parameter Uncertainties The staff’s parameter uncertainty review addresses whether the licensee has: Demonstrated an adequate understanding of the effect of parameter uncertainties (i.e., the significance of state-of-knowledge correlation) and Provided adequate justification for the proximity of the PRA risk metric results to the acceptance guidelines. Since both parameter and model uncertainties may affect the application’s risk results, the staff may refrain from rejecting the application as a risk-informed application on the basis of the comparison of the mean value derived from the propagation of parameter uncertainties, until the review of the licensee’s treatment of model uncertainties has been performed. Page 67 of 83 Slide 16F - Model Uncertainties The staff’s model uncertainty review addresses whether the licensee has: adequately identified key sources of model uncertainties and related assumptions; demonstrated an adequate understanding of the effect on the risk metric results from key model uncertainties or related assumptions; and provided adequate justification for the acceptability of the comparison between the risk metric results and the acceptance guidelines such as qualitative arguments that the sensitivity studies that challenge the guidelines may be considered compensated for by plant practices not accounted for in the PRA model The staff reviews the application’s technical bases and sensitivity analyses to determine whether the licensee has provided adequate justification for the acceptability of the effect of key sources of model uncertainties and related assumptions. Page 68 of 83 Slide 16G - Adequate Justification The staff determines whether adequate justification has been provided by reviewing the licensee’s argument for the acceptability of the application, and may include, but is not limited to, a review of the application technical bases supporting analyses, compensatory measures or monitoring requirements, and other qualitative considerations. Page 69 of 83 Slide 17A - Aggregation Treatment 17 How are different levels of detail, conservatisms (e.g. aggregation) in the PRA treated? The process for treating different levels of detail and conservatisms in the PRA involve the following: Contributors are Combined -- meaning that for all applications, it is necessary to address all applicable hazards and plant operating states when quantifying the risk metrics. Different Levels of Detail and Conservatisms -- addresses the fact that when combing the results of PRA models, the level of detail may differ from one hazard to the next or among the plant operating states, with some being more conservative than others. Biasing of Results -- discusses the idea that significantly higher levels of conservative bias can exist in the level of detail and scope which may introduce varying levels of realism in the corresponding PRA results. Proximity to Acceptance Guidelines -- details how the process of aggregation can influence the proximity of the risk result to the acceptance guidelines and therefore influence the risk-informed application. Page 70 of 83 Slide 17B - Contributors Are Combined For all applications, it is necessary to consider the contributions from the applicable hazards and/or plant operational states when quantifying the risk metrics such as core damage frequency, large early release frequency, or an importance measure. Because the hazards and plant operating states are independent, addition of the contributions (i.e., aggregation of the results) is mathematically correct. However, issues such as varying level of detail in the different hazard or plant operational states analyses should be considered when combining the results from different hazards and plant operational states. Page 71 of 83 Slide 17C - Different Levels of Detail and Conservatisms When combining the results of PRA models for several hazards (e.g., internal events, internal fires, seismic events) as required by many acceptance guidelines, the level of detail may differ from one hazard to the next with some being more conservative than others. The level of modeling can vary even within a hazard such as an internal event, at-power PRA. For example, the evaluation of room cooling and equipment failure thresholds can be conservatively evaluated leading to a conservative time estimate for providing a means for alternate room cooling while other evaluations are more realistic (e.g., defining the onset of core damage for a boiling water reactor occurring at 2’ above the bottom of the active fuel rather than at core uncovery which would be considered a more conservative evaluation). Moreover, at-power PRAs follow the same general process as used in the analysis of other hazards, with regard to screening: low-risk sequences can be modeled to a level of detail sufficient to prove they are not important to the results. Page 72 of 83 Slide 17D - Biasing of Results -Significantly higher levels of conservative bias can exist in external hazards, low power and shutdown, and internal fire PRAs. These biases can result from several factors, including the unique methods or processes and the inputs used in these PRAs as well as the scope of the modeling. For example, the fire modeling performed in a fire PRA can use simple scoping models or more sophisticated computer models or a mixture of methods and may not mechanistically account for all factors such as the application of suppression agents. Moreover, in an effort to reduce the number of cables that have to be located, fire PRAs do not always credit all mitigating systems. To a certain level, conservative bias will be reduced by the development of detailed models and corresponding guidance for the analysis of external hazards, fires, and low power and shutdown that will provide a similar level of rigor to the one currently used in internal events at-power PRAs. However, as with internal events at-power PRAs, the evaluation of some aspects of these other contributors will likely include some level of conservatism that may influence a risk-informed decision. -The level of detail, scope, and resulting conservative biases in the PRAs may introduce varying levels of realism in the corresponding PRA results. In addition, because the phenomena to be taken into account can be complex, the impact of model uncertainties on the PRA results can be larger for different risk contributors. However, these concerns do not preclude the aggregation of results from different risk contributors; but it does require that the decision-maker understand the differences in the confidence with which the significant contributors to the risk metrics are representative of the associated risk. Page 73 of 83 Slide 17E - Proximity to Acceptance Guidelines The process of aggregation can be influenced by the type of risk-informed application. For example, it is always possible to add the core damage frequency or large early release frequency, or the changes in core damage frequency or changes in large early release frequency contributions from different hazards for comparison against corresponding acceptance guidelines. However, in doing so, one should always consider the influence of known conservatism when comparing the results against the acceptance guidelines, particularly if they mask the real risk contributors (i.e., distort the risk profile) or result in exceeding the guidelines. If the acceptance guidelines are exceeded due to a conservative analysis, then it may be possible to perform a more detailed, realistic analysis to reduce the conservatism and uncertainty. For applications that use risk importance measures to categorize or rank structures, systems and components according to their risk significance (e.g., revision of special treatment), a conservative treatment of a hazard can bias the importance measures calculated for that hazard. Moreover, the importance measures derived independently from the analyses for different hazards cannot be simply added together. An integration of different risk models would not provide a good set of integrated importance measures due to impact of different levels of conservatism in the various hazard analyses. Thus, risk-informed decisionmaking based on the use of importance measures will have to consider the uncertainty of the values from different hazard assessments. Page 74 of 83 Slide 18A – Treatment of Uncertainties 18 How are the PRA uncertainties treated in risk-informed decisionmaking? The staff review of a risk-informed application begins with determining the proximity of the application risk metric results to the acceptance guidelines and the licensee’s justification. The justification needed to demonstrate the acceptability of a given risk-informed application, and therefore the extent of the staff review, is largely dictated by the proximity of the risk metric results to the acceptance guidelines. In Regime 1, the risk metric results are well below the acceptance guidelines. In Regime 2, the risk metric results are closer to, but do not challenge the acceptance guidelines. In Regime 3, the risk metric results challenge the acceptance guidelines. In Regime 4, the risk metric results clearly exceed the acceptance guidelines. For each regime, NUREG-1855 provides guidance on considerations for decisionmakers when the risk metric results fall in that regime. Associated EPRI report 1026511 provides guidance to licensees on what information to provide decisionmakers for the respective regimes in order to support that decisionmaking. Link to EPRI Report 1026511: EPRI Report 1026511, “Practical Guidance on the Use of Probabilistic Risk Assessment in Risk Informed Applications with a Focus on the Treatment of Uncertainty,” [https://www.epri.com/#/pages/product/000000000001026511/] EXIT: This link is external to the NRC website Page 75 of 83 Slide 18B - General Justification The justification for a given application should be commensurate with the proximity of the risk metric results to the acceptance guidelines. In general, more justification will be needed for a given application when the risk metric results are closer to challenging or exceeding the acceptance guidelines than when the risk metric results are further below the acceptance guidelines. Page 76 of 83 Slide 18C - Regime 1 In Regime 1, the risk metric results are considered to be well below the acceptance guidelines when the mean value of the risk metric results are less than the acceptance guidelines by approximately one order of magnitude or more. The staff would look for an assessment which demonstrates that the state-of-knowledge correlation does not affect the risk metric results of the PRA, would also evaluate the application to determine whether the validity of the assumptions made in the application PRA will be appropriately monitored via the implementation of specific measures and criteria, would look to see whether degraded performance can be detected in a timely fashion and would likely place little importance on the licensee’s use of compensatory measures, depending on the justification that is provided. Page 77 of 83 Slide 18D - Regime 2 In Regime 2, the application PRA risk metric results do not challenge the application acceptance guidelines and the mean value of the risk metric results are approximately within an order of magnitude of the acceptance guidelines. The staff would look for an assessment which shows that the state-of-knowledge correlation does not affect the risk metric results of the PRA, would examine the application to ensure that the proposed performance monitoring is appropriate and adequate for the application and whether degraded performance can be detected in a timely fashion. The staff would examine the peer review (as discussed in Regulatory Guide 1.200) findings with a higher degree of scrutiny than for applications that fall into Regime 1 so as to better understand how particular findings were resolved as well as the general impact of the findings. In general, it is unlikely the staff would perform an audit on the application PRA for those applications that fall into Regime 2. Link to Regulatory Guide 1.200: Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities [https://www.nrc.gov/docs/ML0904/ML090410014.pdf] Page 78 of 83 Slide 18E - Regime 3 In Regime 3, the risk metric results are considered to challenge the acceptance guidelines when the mean value of the risk metric results are either slightly less than or slightly greater than the acceptance guidelines. The staff expects that a quantitative assessment would be provided which shows that the state-of-knowledge correlation does not affect the rick metric results of the PRA. The staff would examine the application to ensure that the proposed performance monitoring is adequate and determine whether degraded performance can be detected in a timely fashion, would expect compensatory measures to be in place, would examine the peer review (as discussed in Regulatory Guide 1.200) findings with an even higher degree of scrutiny so as to better understand how particular findings were resolved as well as the general effect of the findings, would likely consider the parameter uncertainties and the identified key sources of model uncertainties to determine if greater depth of review of the uncertainty aspects of the application should be performed, would likely perform an audit of the application PRA to determine the cause of the change in risk and would generally be limited to an investigation of the significant issues in the PRA and the key sources of model uncertainties. Link to Regulatory Guide 1.200: Regulatory Guide 1.200, “An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities [https://www.nrc.gov/docs/ML0904/ML090410014.pdf] Page 79 of 83 Slide 18F - Regime 4 In Regime 4, the risk metric results are considered to significantly exceed the acceptance guidelines when the mean value of the risk metric results are more than just slightly greater than the acceptance guidelines. The staff would typically not accept such applications unless the application represents an overall reduction in plant risk. The staff would expect the licensee to identify those aspects of the PRA that are conservative and bounding and then perform a more realistic assessment to support the application, would look for a quantitative assessment which shows that the state-of-knowledge correlation does not affect the rick metric results of the PRA, would examine the application to ensure that the proposed performance monitoring is adequate and determine whether degraded performance can be detected in a timely fashion, would determine the appropriateness of the compensatory measures, would thoroughly review the licensee’s PRA peer review findings, would review in greater detail the parameter uncertainties and the identified key sources of model uncertainties and their effects on the risk metric results, would perform a more in-depth audit of the application PRA. Page 80 of 83 Slide 18G – Application Acceptance/Rejection If accepted by the staff, the risk-informed application is considered to have (1) an acceptable treatment of uncertainties, and, (2) to have met the fourth risk-informed decisionmaking principle of posing an acceptable risk effect to the plant. Conversely, if the staff rejects the application, the risk-informed application is considered to have 1) an unacceptable treatment of uncertainties, or 2) poses an unacceptable risk effect to the plant. Page 81 of 83 Slide 18H – Application Acceptance In many cases, a risk-informed application requires consideration of the risk effect from multiple hazards and plant operational states and, as such, the risk contribution from each analysis must be combined, or aggregated, into a single risk metric, such as core damage frequency or large early release frequency. When combining the results of PRA models for several hazards (e.g., internal events, internal fires, seismic events) as required by many acceptance guidelines, the level of detail and level of approximation is likely to differ from one hazard to the next with some being more conservative than others. Often conservative bias can be larger for external hazards, internal fire, and low power and shutdown risk contributors, and can be one reason why the associated uncertainties can be larger. Therefore, the staff determines whether the licensee demonstrates adequate understanding of both the level of detail associated with the modeling of each of the hazards as well as the hazard-specific parameter and model uncertainties. This involves determining whether the licensee demonstrates an adequate understanding of: 1. The individual risk contributions and the parameter uncertainties associated with the analysis of each hazard or plant operational state and 2. The sources and effects of conservatisms and model uncertainties that significantly affect the application results. Page 82 of 83 Course Completion Slide Congratulations! You have completed this course and should now have a good understanding of the guidance presented in NUREG-1855 and the Associated EPRI Reports. For training on how to implement the guidance in these documents, please sign up for the training workshop or course that goes into detail, using various examples of riskinformed activities in order to illustrate application of the guidance. Page 83 of 83