Chapter 8 Operational Risk Tools - Scenario Analysis PDF

7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing Chapter 8 Back Up Book for Printing Chapter 8: Operational Risk Tools - Scenario Analysis Learning outc...

7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing Chapter 8 Back Up Book for Printing Chapter 8: Operational Risk Tools - Scenario Analysis Learning outcomes and assessment criteria 8. Understand the nature and role of scenario analysis in the management of operational risk. 8.1 Examine the nature of scenarios. 8.2 Describe the benefits of scenario analysis. 8.3 Explain the internal and external factors which may affect the scenario analysis process. 8.4 Describe the approaches to analysing scenarios. 8.5 Examine the challenges associated with different approaches to analysing scenarios. 8.6 Describe the elements involved in constructing scenarios. 8.7 Describe the forms of bias which may affect scenario analysis. 8.8 Explain the methods of validating scenario analysis results. 8.9 Explain the relationship between scenarios and other operational risk tools and techniques. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 1/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing Key themes The key themes of this chapter are as follows: Basic concepts relating to scenarios. Objectives and benefits of using scenarios. The different uses of scenarios including challenges. How to identify where scenarios are required, and how to document a scenario. Factors which affect scenario analysis outcomes and where scenarios fit into the operational risk framework. Introduction to Chapter 8 A scenario can be defined as an outline, description or model of a combination of unexpected or adverse events. The scenario should be ‘severe but plausible’. Firms typically aim to develop a suite of scenarios that offers a range of different severities. Scenarios are typically described using event types or risk categories. They may detail the causes and potential impacts of the event, should it actually crystallise. Analysis of the causes may lead to action to mitigate the outcome of the scenario through management actions. Scenarios can be maintained in a library or repository and re-used or adapted to reflect changing circumstances. The scenario is the input into a scenario analysis exercise and is intended to assess the firm’s exposure to that scenario over time. Generally, information from different aspects of an operational risk framework can be included in a scenario analysis: a description of the event, an inventory of relevant risks and controls, relevant internal and external loss data, risk indicators that may describe or even predict the scenario event, and use of the scenario for risk capital calculation purposes. Scenarios also typically include a causal analysis, along with expected direct and indirect impacts, particularly those of a reputational nature. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 2/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.1 Examine the nature of scenarios A scenario is essentially a story or model of a possible sequence of unexpected or adverse events. For example, a cyber-attack at a retail bank, resulting in compromise of customer accounts and regulatory sanction for inadequacies in systems and controls, or a country wide flu pandemic resulting in significant disruption to employee availability including loss of key senior decision makers - if this was at a life insurer this could be at a time of increased customer demand. Further examples are also referred to at various points throughout this chapter. Firms and individuals examine the scenario and then assess their exposure should that scenario manifest itself as an actual event. As such, scenarios are considered a powerful forward-looking tool or technique. 8.1.1 Objectives of a scenario analysis These are some generic objectives for conducting scenario analysis: Obtain consensus of the magnitude of the firm’s exposure to severe but plausible adverse events or risks forming the scenario. Identify specific exposures where the firm is either inadequately prepared to deal with the situation or where the current control environment is insufficient to prevent or mitigate the problem. Evaluate medium- and longer-term exposures to particular risks, either for capital estimation or loss provisioning purposes or for inclusion in the firm’s risk register. Evaluate factors relating to new business activity, business partners, vendors, products, locations or business plans, in order to fully understand the risks and exposures involved; and Comply with regulatory requirements or the directions of authorities in evaluating particular exposures, often as part of industry-wide assessment. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 3/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.1.2 Types of scenarios Scenarios are multi-variable occurrences which capture the impact of several events at the same time or the occurrence of a chain of linked events. For example a serious system failure at a critical point in time, alongside an urgent need to process a strategically important transaction, alongside senior decision makers being unavailable for a critical period. One issue with multi-variable scenarios is where to establish the boundaries – the greater the number of variables in a single scenario, the more likely the output will be unrealistic and hard to reproduce consistently. Different assessors will place a different emphasis on different individual components of the scenario, which may result in widely diverging opinions on the implications of that scenario for the firm. Nevertheless with careful design and validation it is possible to come up with scenarios that are reproducible and plausible (albeit severe). There are different sources from which scenarios can be identified and selected. Some sources are internal to the firm itself, including some driven by the business based on actual exposures they have. These are termed bottom-up scenarios. Other sources are more macro-level and determined either at senior management or governing body level or by group functions. These are termed top-down scenarios. Typically bottom-up scenarios are valuable for assessing exposure, measuring the firm’s response to the scenario, and identifying potential gaps in the firm’s control environment. Whereas top-down scenarios are more commonly used by senior executives in response to regulatory directives, industry events, to inform strategic planning, or as part of a firm’s capital estimation process. Often different scenarios are driven by the type of business or products offered in different areas of a firm: for example in global markets businesses a rogue trading scenario and its knock-on effects is a commonly-used scenario; whereas in asset management you might see a scenario based on a serious fiduciary breach and its consequences; or in retail banking a co-ordinated cyber-attack on customer accounts and its associated impact. Some more common or generic scenarios lend themselves to being assessed across the entire firm, for example the consequences of key people leaving the firm, core system outages, a natural disaster or a terrorist attack. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 4/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.1.3 Number of scenarios The question often arises as to how many scenarios a firm should have. As is often the case there is no right number. It depends on the individual firm and its business model, the environment in which it operates and how scenarios are used in the firm. The number of possible theoretical scenarios a firm has in its ‘scenario library’ is one thing. But in terms of actually analysing the impact of different scenarios, that number may be much larger after it has been multiplied up by the number of instances it is used, by different business lines, in different geographical locations, or by different legal entities. Workplace reflection Does your firm have its own internal library of scenarios? If so, how many actual scenarios does it contain? If the firm does not have a single library of scenarios, check with functions such as risk management, IT security, strategic planning, finance and business continuity planning and see whether any of these areas have their own libraries of scenarios, then consider developing a business case to centralise these into a common scenario library. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 5/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.2 Describe the benefits of scenario analysis Although the use of scenarios is a mandatory element of capital estimation under the AMA approach to operational risk capital estimation, firms engage in scenario analysis for a range of reasons, not just capital estimation. These are some of the benefits of a well-constructed and comprehensive scenario analysis programme: Fosters risk management awareness throughout the organisation and helps secure senior management engagement. Helps embed a strong risk culture in the business and the firm’s governing body. Helps identify new strategic and operational challenges emerging in the business environment and provides forward-looking assessments of these risks. Identifies and supports the development of controls and risk mitigation strategies, business and contingency planning, and internal and external reporting procedures. Informs the setting of the firm’s risk appetite and tolerance for losses. Feeds into enterprise-wide capital planning, measurement and allocation, and helps fulfil other regulatory requirements. Identifies where the firm may require specific forms of insurance, providing a mechanism for fair valuing any insurance considered and supporting existing insurance renewals; and Supports training outcomes, in that staff can be exposed to, and prepare for, potential adverse issues and situations they may face, even if the scenario is more extreme than most people’s experience to date. 8.3 Explain the internal and external factors which may affect the scenario analysis process Scenario analysis is, by its nature, subjective. It is based on estimates made by subject matter experts about the potential implications of an extreme but plausible future event. While this subjectivity is accepted as an inherent feature of scenario analysis, it can be mitigated by careful design and consideration in three areas. 8.3.1 Purpose The purpose of the scenario analysis will affect its process, content, approach, participation and output. If the purpose is made clear upfront these features can be calibrated to ensure the analysis is both efficient and useful, rather than relying on a ‘one size fits all’ approach and running the same scenario for all purposes. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 6/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.3.1 Purpose 8.3.2 Participants Selection of the right participants to help with the scenario analysis programme is important to ensure the right level of knowledge and experience to enable them to contribute meaningfully. For example, at business entity level, a person with broad business experience is desirable, ideally with knowledge of business plans, budgets, historical performance and operating information (e.g. headcount, transaction volumes, customers). Participation by the individual in the business’s management process and meetings is preferable, and knowledge of business resiliency criteria and plans for the business entity would be beneficial. For support functions, the individuals identified should be subject matter experts in whatever area they represent. Participants should come from a variety of functions to provide different perspectives on the scenario being considered. 8.3.3 Business environment The business environment, both in the firm and externally, can influence scenario analysis outcomes. A negative business or economic environment involving, for example, a recession, increasing inflation, or a business cost-cutting programme, can affect participants’ judgment and make them more pessimistic about outcomes. It is important to remember that the business environment changes over time and as a result the same participant assessing the exact same scenario but under different business environmental conditions can be expected to generate different responses. This can affect the consistency of scenario output over time. It is also a good reason for scenarios being regularly reviewed. To address these issues some firms do a business environment factor assessment before the scheduled scenario analysis, whereby collective views on the business environment are collated in a document which sums up the agreed business environment and assumptions. This approach aims to ensure at least that all participants have the same reference point for the purposes of the scenario analysis. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 7/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.4 Describe the approaches to analysing scenarios 8.4.1 Factors in deciding on an approach There are two factors to consider when deciding on which scenario analysis approach to take: the purpose for which the analysis data will be used, and the level and availability of participant. Purpose of the analysis: Where the assessment is to establish the exposure to the scenario, determine the likely response from the firm and to identify controls gaps which need to be corrected, approaches where participants co-operate with each other to determine the outcome are beneficial, with examples being war gaming, workshops and research and verification techniques. If bias is to be avoided as far as possible, as is usually required for capital measurement purposes, approaches which isolate participants and obtain individual views are preferred, for example, using interviews, Delphi methods (see 9.4.2 Different approaches) and online questionnaires. Participants: If the participants are very senior, have limited availability or are located at remote offices, it may be best to use online questionnaires, short interviews or Delphi methods. More details on all of these techniques are set out below. For more junior staff, or participants who have restricted knowledge (e.g. specialist subject matter experts whose knowledge and experience is restricted to just one area) – or where large numbers of participants have to be involved – workshops are more suitable. In determining an approach, an objective to consider is to try to avoid the risks of one individual, or a small number of individuals, exerting too much influence on the process, resulting in biased outcomes. Individuals may even seek to game the process in some cases, for example if there is an important capital outcome which significantly benefits their business. To minimise the risk of these kind of outcomes, online questionnaires, Delphi methods and individual interviews may be the most appropriate approaches for particularly sensitive or important scenario analysis programmes, for example those used as an input to capital allocation or other regulatory processes. Learning activity Research the approach to scenario analysis used in your firm and think about the reasons why that approach might have been selected. Compare your firm’s internal approaches to those described in this Workbook. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 8/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.4.2 Different approaches A firm is likely to use a number of different approaches for different scenario analysis purposes. Firms will also design their programmes iteratively, and use different approaches in different iterations. For example, in round one, using interviews or online questionnaires to obtain every individual’s perspective, and consolidating the data. Then, in round two, using workshops to confirm and finalise the results. It is even possible to then have a third round of iteration, using a war gaming method to develop corrective or remedial activities. Workshops: These are the most commonly used mechanism to gather scenario analysis outcomes. They require a range of participants to meet, usually under the guidance of a facilitator, to discuss the scenario and reach a collective view. Even if participants have to familiarise themselves with the scenarios ahead of time, the facilitator should introduce the scenario at the start of the workshop and ensure a common understanding before starting the analysis. Workshops can be open to bias, particularly if a participant with a strong view is able to force that view on other participants. Workshop outputs are also prone to cultural differences and thus may not work equally well in all geographical regions. Interviews: This approach needs strong facilitation skills, with a trained interviewer conducting one-on-one interviews with each participant. This approach is effective in countering bias or gaming, and is very suitable for senior participants, but is resource intensive. Multiple iterations of the process may be required if participant responses vary considerably, but it is possible to counter this by using follow-up workshops after the initial interview. Online questionnaires: This approach requires access to an online tool which documents the questions, distributes questionnaires to participants and provides controlled collection of responses. While tools such as email can be used, they do not offer the degree of control that a specialist tool provides. Essentially, this approach allows participants to complete their assessments at their own convenience. It therefore requires buy-in and discipline from participants to actually complete the process. This approach may also include multiple iterations, asking participants to confirm previous responses or to consider factors which may previously not have been considered. It is another way to minimise bias although the facilitator has no control over participant action outside of the tool. Delphi method: This is a scientific method which uses a panel of experts who answer specific questions about a given scenario. Experts answer questions in two or more rounds. After each round, the results are analysed and then used as input into a further round of questions. The process summarises not only the responses, but also the reasons for the judgement, and encourages experts to revise their earlier answers in light of the replies of other experts, with the overall process seeking convergence of expert opinion. Usually participants remain anonymous, allowing free expression and preventing some participants from dominating the process. While scientific, this approach depends on the firm having available a supply of subject matter experts who can be represented on the panel. Furthermore, in spite of best intentions, the method can lead to diverging opinions and disagreement on outcomes, rather than always promoting consensus views. So it has to be used with caution, but can be a useful supplement or alternative to more traditional approaches. Research and verification: This approach involves a central team undertaking research into the scenario – either internally, externally or in combination – then presenting participants with a range of actual historical data. This can be done via a workshop, interviews, or an https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 9/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.4.2 Different approaches online questionnaire. The aim is to ask the participants to verify that the results represent an extreme but plausible outcome for the scenario in question. An extension of this approach asks participants to then suggest a further worst possible outcome, to make it more forward looking. This approach can be criticised for ‘anchor bias’ (see section 8.7. below), but is sometimes used in scenario analyses where a clear historical dataset exists as a basis for forward looking projections. External data: This approach minimises the time impact on participants, but can be criticised for not being sensitive enough to the internal workings of the firm, in particular differences in size, business models and control environments. Essentially, having identified the areas of concern for which scenarios are required, an analyst researches external loss data – from either a data consortium or publicly available sources – to obtain a severe but plausible impact for that area of concern. Frequency of occurrence can also be analysed using this method. The results are then used as the basis for analysing the firm’s exposure internally. War games: The immediate goal of a war game is to anticipate competitive developments – or knock-on impacts from an event occurring – and then to formulate viable options in response to the emerging scenario. Participants in a war game usually work in teams against each other, are provided with evolving situations and information, and have to establish strategies, tactics and control mechanisms to defend their position, objectives or plans. War games typically involve a lengthy debrief process, involving neutral observers who have noted results throughout the process. By their nature, war games are time- and resource-intensive, and more suited to ad hoc analysis: they do not readily support consistent reassessment of a scenario over time. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 10/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.5 Examine the challenges associated with different approaches to analysing scenarios Challenges that the scenario programme will need to address and overcome include: Getting the right people involved in assessing scenarios. This includes finding good facilitators who have business knowledge and skills, have good interpersonal skills suitable for facilitating workshops and interviews and having the knowledge and ability to manage bias and gaming is a difficult task. Getting participants to ‘think the unthinkable’. Comments such as ‘I have no experience’, ‘that has never happened before’ and ‘that is not my responsibility’ are typical responses when a participant feels uncomfortable in responding to a particular scenario. They may need to be guided to express an opinion without introducing any bias or leading them to an answer. At the other extreme, scenarios must remain plausible. Participants' creativity can sometimes require management by the facilitator to ensure an appropriate balance between extreme but plausible is obtained. Ensuring common understanding among participants of what the scenarios are about, and if appropriate making sure the outputs are consistent over time. Language, culture and terminology all play a role in shaping a participant’s understanding of what they are being asked to evaluate and respond to. Without a common understanding, the response data will not be comparable. Or the same participant may provide different responses to the same scenario at two different points in time. For capital modelling estimates, converting subjective measures into objective data parameters, and applying filters to exclude complete outliers is challenging. A way to address this challenge is to design the data to be collected during the scenario analysis process specifically so they can be usable for modelling purposes. For example, collecting a median impact in addition to the mean may assist the modelling team to develop estimates of distributions. A final challenge is obtaining regulatory acceptance of the scenario analysis programme, particularly if the scenario assessment output is to be used in capital estimation. This will require the firm satisfying its regulator that these challenges, particularly the ones related to completeness, consistency, bias and gaming, have been adequately addressed. It follows from all of the above that a comprehensive, well-organised scenario analysis programme will require a reasonable financial budget, not just for the staff who manage the programme, develop the scenarios and facilitate the assessment sessions, but also in terms of time investment by management and participants. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 11/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.6 Describe the elements involved in constructing scenarios Having determined the purpose behind its scenario analysis – and having established the method by which the analysis will be collected – the next step is identifying what kinds of scenarios the firm wishes to analyse, and thus which elements it needs to construct them. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 12/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.6.1 Identifying the areas to be covered by the scenario The purpose behind the scenario analysis process is important to bear in mind when deciding this. For example, if the purpose of the scenario analysis is to fill in areas where little or no event data exists as an input into a capital estimation process, then the identification process will be focused solely on filling in those blank spaces. Conversely, if the objective is to identify exposures where the firm may be inadequately prepared to deal with the situation, the identification process will tend towards looking for ‘unknown unknowns’. Some common approaches to scenario identification include: Loss event driven approach: This approach assesses material historical losses, both within the firm and amongst peer firms in relevant geographies, then uses the identified losses as the basis for selecting which exposures to assess. Risk-driven approach: This approach focuses on identified risks facing the firm, using the output of the firm’s RCSA programme, as set out in Chapter 5. Control-driven approach: This approach focuses on identified key controls and the impact of the failure of such controls. Expert opinion: This approach relies on experts’ suggestions of what scenarios should be assessed. While expert opinion may be used as an input, undue reliance on expert opinion not only introduces the potential for bias and gaming, its effectiveness is significantly reduced by knowledge gaps, effectively the boundaries of the expert’s knowledge. Industry standards: Where relevant industry groups select or advocate specific scenario sets, these should be considered by the firm. However, it should be acknowledged that these tend to be scenarios that affect all industry participants equally, and will not take into account exposures specific to the firm or its products, services and culture. Regulatory requirements: Periodically, regulators and other authority bodies may impose or advocate specific scenarios to be analysed by the industry at large or by individual participants. Research: In this approach responsible staff research emerging themes, surveys of top risks, specific regulatory focus areas and other industry sources to identify areas of concern, then identify scenarios relevant to the firm in such areas. Learning activity Draw up a manifesto of all the approaches used across your firm to identify what scenarios are to be assessed. What are the benefits and shortcomings of these approaches, and how might they be rationalised? The next step is to identify which specific scenarios the firm will examine. For example, if the firm has identified rogue trading as an area of concern for its global markets business, it will have a number of different elements to consider including in its scenarios. For example: Concealment of executed trades. Recording of fictitious trades. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 13/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.6.1 Identifying the areas to be covered by the scenario Disregard for trading limits. Manipulation of revaluation rates and prices. Market volatility. It may be useful at this point to make use of subject matter experts and to consider existing key controls, to exclude scenarios which are so unlikely that they justify exclusion based on the ‘extreme but plausible’ definition. Once that has been done, any remaining candidate scenarios should be carefully evaluated for those which are less unlikely to manifest themselves, and then prioritised accordingly. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 14/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.6.2 Constructing the scenario Information to include in the documentation of a new scenario include the following: A clear, unambiguous and descriptive name. A detailed description of the scenario and, if appropriate, of individual variables or components that comprise it. The time horizon over which the scenario may manifest itself. Qualifications and conditions as to what is in scope for the scenario and what is excluded from the scope. This may include, for example, restrictions on processes, products, geographic locations, client types or distribution channels. Direct and indirect impacts which might occur in the scenario. Causal factors which may give rise to the scenario, or which have to be present for the scenario to occur. This basic information can be supplemented with other data to enrich the description of the scenario and make it more meaningful to scenario analysis participants: Controls which may be in place and should be taken into consideration. Risk or control indicators that may provide advance warning of changes in exposure. Business environment factors that may determine whether and how the scenario manifests itself. Historical internal loss data, consortium loss data and/or publicly-sourced loss data relevant to the scenario. Previous scenario analysis results. Learning activity For each of the scenario uses you think your firm may have a need, evaluate each of the suggested pieces of information you would include for each scenario use. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 15/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.6.3 Outputs from the scenario analysis The next step is to identify the outputs required from participants in the scenario analysis. Again, the use for which the scenario is intended will impact what information is collected. It is advisable to consult other interested stakeholders who will be using the output in order to ensure their needs are also included. For example, if a scenario is being used to assess a firm's capital requirement, it is important that all the data needed to feed the capital model are collected during the scenario analysis process. Frequency: This measures the number of times the scenario may manifest itself within the analysis time horizon. For example, a utility outage could occur once each year over a ten year horizon. Likelihood: As defined in Chapter 5, likelihood is the possibility of something happening. It can be expressed in a number of ways, but is commonly conveyed by ranges of values representing a low, medium or high likelihood of occurrence. E.g. low likelihood: less than 1 in 10 years; medium likelihood: 1 in 1-10 years: high likelihood: 1 in a 12- month period. Probability on the other hand, refers to the calculated chance of something occurring based on quantitative parameters, data or a mathematical process. It is important to be absolutely clear which output you are looking for (in most cases likelihood rather than probability) and to use the correct terminology with participants. Impact and severity: These terms are sometimes used interchangeably and can include direct or indirect, and financial or non-financial. As discussed in Chapter 5, for RCSAs impacts are generally quantified less precisely using ranges on a risk matrix for example 1-5 or Low- High rating. For scenario analysis, impacts will generally be translated to a specific currency impact, particularly where these are to be used for capital estimation purposes. Impacts can be estimated at different levels of severity such as ‘worst case’ (which measures the worst observed – or historical – manifestation of the exposure over a given time horizon) and ‘extreme case’ (which looks to measure an even higher confidence interval than the observed case). 8.6.4 Documentation Once the scenarios have been documented, the details should be collated and distributed to participants prior to the analysis process. Many firms store their scenarios in a ‘library’ for future re-use. When a scenario is re-used, the information identified in the sections above should be reviewed and brought up-to-date, especially in relation to quantifiable data such as historical losses or risk metrics. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 16/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.7. Describe the forms of bias which may affect scenario analysis 8.7.1 Bias and gaming A common concern raised against the use of scenarios relates to their subjectivity and dependency on ‘expert opinion’ and hence the possibility of undue influence or potential for response bias to develop an incorrect representation of the true exposures facing the organisation. A robust scenario analysis programme seeks to minimise the potential influence of these kinds of factors – especially in cases where the scenario analysis outputs may be used for capital measurement purposes, either under a PiIlar 1 capital modelling approach (e.g. AMA) or for Pillar 2 capital purposes, as set out in Chapter 9, The Regulatory Treatment of Operational Risk. There are various ways in which these issues can be addressed and their impacts minimised. Clear detailed scenario documentation is one way. Using different tools for different purposes is another. For example, it is good practice to employ one-on-one interviews for analysing capital measurement scenarios, alongside workshops for operational management scenarios. Bias can be defined as an inclination of temperament or outlook to present or hold a partial perspective, often accompanied by a refusal to consider the possible merits of alternative points of view. There are many forms of bias, but three classes of bias in particular are relevant for scenario analysis: Subconscious or judgmental bias. Extraneous bias. Conscious or motivational bias. Subconscious or judgemental bias arises from: Information provided to participants. Participants’ frame of reference. The inability of participants to process or grasp the issue at hand. Memory limitations. Anchoring the starting point against known or given facts. Tunnel vision: focusing only on one or two aspects of the issue and not the full spectrum. Extraneous bias arises from: https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 17/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.7. Describe the forms of bias which may affect scenario analysis The manner in which scenario assessment is facilitated, including the structure and sequencing of questions. The order in which relevant information is provided to participants. The attitude and body language of the facilitator. Interruptions or other distractions, such as intrusion into sessions by third parties, phone calls or other commitments. The overall risk culture of the firm and the way in which the importance of the scenario analysis programme has been communicated from the top down. Conscious or motivational bias arises from: Risk managers or control managers who have an interest in influencing a result from their own point of view. ‘Herd instinct’: not wanting to have a different opinion to other participants. ‘Lead bull’: a dominant individual’s opinion or direction. Deliberate gaming. A perceived link between the scenario impact and capital estimation and allocation – hence a desire to keep the impact as low as possible. Concealment of truth. The following forms of bias are of particular concern: Anchor bias: A form of subconscious bias where participants pick up on some initial estimate or statement provided by someone else, then use that as the basis for their own estimate. The source of the anchor may be a comment made by the facilitator or another participant, or illustrations of previous internal losses or external losses suffered by other organisations. Culture: An extraneous bias which has particular implications for firms that are internationally active. Not all national (or organisational) cultures have the same attitude to disclosure, openness, possible failure or the potential for losses to occur, thus causing the participant to understate their estimates. Similarly, local language and even regional terminology can compound cultural differences and may lead to misunderstanding or mis-interpretation of what is being assessed. Dominance: A motivational bias, where a very dominant participant or the presence of ‘the boss’ may inhibit challenge or may cause certain participants to abstain from constructive participation in order to avoid conflict. It is closely linked to ‘herd instinct’ bias, where participants will rather agree with the dominant participant than be considered as an outsider or not a team player. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 18/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.7. Describe the forms of bias which may affect scenario analysis Extroverts: Another form of motivational bias, this derives from individuals who have a lot to say and an opinion on everything and unduly influence others by virtue of being the most vocal presence. This is sometimes a deliberate ploy on the part of the vocal individual to divert attention from their area of responsibility to somewhere else. Gaming and ulterior objectives: Participants may deliberately adopt a specific stance simply to avoid the truth emerging or to achieve other ulterior motives or objectives. An example would be significant overstatement of the potential impact of technology failure, in order to manipulate future IT investment or even to settle a score with IT. Learning activity Construct a list of your work colleagues who have recently participated in some group activity. Against each participant, indicate whether they were passive or active in participant and whether they were a leader or a follower in the proceedings. Ask yourself why they adopted the stance they did on any particular issues which arose. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 19/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.7.2 Correlation and linkages As we said in the introductory section to this chapter, operational management scenarios take into account additional elements that might be correlated, linked or sequential from the initial trigger event. For example a linked operational scenario might be an earthquake in Turkey triggering an earthquake in Greece at a time of peak operational volume, requiring concurrent business management. More generally it is well-understood that risk events lead to or are correlated with other risk events. Looting and theft often occur after a natural disaster. Operational crises often occur at a time when senior management teams are not available for some reason (for example a management offsite) to deal with them. Rogue trader events often lead to regulatory sanctions, fines and loss of reputation. It is important to retain information on these scenario components occurring individually – without knock-on or linked or correlated factors – mainly to ensure that results can be reproduced consistently over time. That said, it is also very important that the participants in an operational management scenario analysis are encouraged to consider a range of potential outcomes, incorporating different combinations of variables into a common core scenario. In addition to understanding correlations and linkages between possible scenario components and thus scenarios themselves, participants should be encouraged to consider the implications of certain firm-specific issues, which may influence their analysis: Geographic footprint: Would a specific scenario affect the entire group, a sub-set of the group or only a single unit within the firm? What impact would this have on the group as a whole? If the impacted unit provides common services across a broad set of business entities, how might they be affected? Business concentration: Where business activities are concentrated in specific locations, for example, London or other capital cities – how is this affected by a geographic scenario such as natural disaster or terrorism? What knock-on effects might need to be included in the scenario due to, say, concentration of staff in the location? Does this increase the potential for correlated impact across the group? Outsourcing or shared services: Where services are outsourced or shared within a group does this introduce different risk factors for consideration? For example does it introduce a concentrated exposure to one business unit or a particular geographic location? Cyclicality: Should the observed cyclicality of events, economic and business cycles be factored into the analysis, in terms of introducing potential linked or sequential outcomes into the scenario? Economic recession tends to result in headcount reduction which in turn results in a lack of preparedness for an upturn in volume when the economy recovers which results in increased documentation errors and legal risk. Note that in a geographically dispersed firm, different locations may be affected differently by the same cycles and may also be affected by different cycles. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 20/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.8 Explain the methods of validating scenario analysis results Given the challenging nature of a scenario analysis programme – and the various factors which can affect the outcome of the scenario analysis – an important part of the process is validation. Once the scenario analysis outcomes have been collated, the firm needs to engage in a validation process to ensure they are appropriate. One way to do this is to compare the results against a known data source, such as the range of losses reported into a loss data consortium or from other public sources. This is a widely used technique for assessing regulatory fines and costs. First a benchmark or comparator firm is identified, ideally of a similar size and business model and operating in the same local markets, which has incurred regulatory sanction and publicly-disclosed fines. Those fines can then be compared to our firm’s scenario outcomes, to validate or adjust our outcomes. A similar approach is possible for assessing the costs of specific regulatory action in particular cases (for example so-called Section 166 reports in the UK), which are often disclosed by regulators in their annual reports and can be used as part of a scenarios pertaining to product mis-selling or regulatory sanction. Another technique used is to plot outcomes against each other in order to highlight any significant outliers, as shown in the example below. For each participant, the typical case impact has been plotted on the bottom axis and the extreme case impact on the vertical axis. The outliers – those values that far exceed the average – are shown as red diamonds, while in this diagram the largest known public loss event in this category is denoted by the orange square. This kind of representation allows decisions to be made to adjust a scenario definition (if an outlier reveals the scenario to be too extreme and not sufficiently plausible) or to exclude particular outcomes on the same grounds. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 21/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.8 Explain the methods of validating scenario analysis results Figure 8.8: Oversight failures in Product Development and Project Management Workplace reflection Which of the outputs shown in the diagram above would you exclude and why? If the purpose of your assessment is for operational management purposes, would you decide differently than if it is for capital estimation purposes? https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 22/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.9 Explain the relationship between scenarios and other operational risk tools and techniques A firm’s scenario analysis programme does not exist in isolation from its other operational risk tools and techniques. One way in which the inter-relationships between the different tools can be illustrated is to focus on the time horizon of each, as shown by the following graphic. Figure 8.9(a): Scenario analysis and related tools As this graphic suggests, scenario analysis is focused more on the medium-to-long term future rather than the short term, working on the assumption that an extreme (but plausible) operational risk scenario will probably take some time to play out. This is based on the fact a scenario, as we have defined it, has multiple variables and probably a sequence of linked events rather than being typically an instantaneous or single factor shock. The other tools in this framework, meanwhile, tend to be focused on the past, the present or the short-term future. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 23/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.9 Explain the relationship between scenarios and other operational risk tools and techniques Unlike other risk types, operational risk is often characterised as showing a ‘log-normal distribution’ of frequency and severity. In other words, when the number of operational risk events is plotted against their impact it shows a large number of events with small impacts, and a small number of events with large impacts. The number of events decreases as the impact increases, to the point where the number of events with a very large impact is very small as shown below: Figure 8.9(b): Operational risk event distribution The distribution can be further divided into two parts, namely ‘expected losses’ and ‘unexpected losses’. Expected losses (EL) are high frequency low severity events which are inherent in the business. Unexpected losses (UL) typically arise from normadverse or unexpected events which are not routinely accepted as an inherent part of business. The different tools also measure different areas of the typical loss distribution, as shown below. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 24/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.9 Explain the relationship between scenarios and other operational risk tools and techniques Figure 8.9(c): Occurrence of expected and unexpected losses Note that “EL” in the diagram above is shown by the area of the curve to the left of the “EL” line. “UL” s thus unexpected losses, covering the remainder of the distribution. UL99.9 represents a high confidence interval, a 1 in 1000 event; whereas UL Business represents a smaller internal confidence interval, say a 1 in 20 event (95% confidence), which is typically more aligned to the way business people view risk on a day-to-day basis. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 25/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.9.1 Risk and control self-assessments One issue which every scenario analysis programme faces is making clear the differences and similarities between risk and control self- assessment (RCSA) and scenario analysis. RCSA was covered in Chapter 5, but the following table sets out a comparison between the two tools: Aspect RCSA Scenario Analysis Identification of the high level risks Risk Management: understand a specific risk in sufficient detail to enable management associated with a specific unit. Primary to be properly prepared to deal with the event. Risk Measurement: generate frequency Objective and severity data points to assist in calculation of capital requirement. Risks are identified and briefly Usually assumes risks have been identified, and builds a ‘story’ from that point which described through the RCSA process. explores what else could go wrong and how management might respond if the event Identification should occur. Analysis of the event is generally bigger picture and, for risk of Risks management purposes, more detailed. May include RCSA results, internal and external loss data, risk indicators and audit Inputs to the May include internal and external loss results. Process data, risk indicators and audit results. https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 26/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.9.1 Risk and control self-assessments Aspect RCSA Scenario Analysis Usually typical case and perhaps worst Risk management: Extreme but plausible case or worst case in a (typically) 10 or 20 case in a 2 year horizon, typically 12 year horizon. Impacts are estimates. Impact months outlook. Impacts are Evaluation estimates. Risk measurement: Impacts are estimates at various points on the loss distribution curve, but primarily focused on the ‘tail’ of the distribution (i.e. extreme points on the Frequency distribution). Evaluation Can be a broader audience, often including senior executives as well as other Business Unit line managers and their groups/departments. Stakeholders daily partners. RCSAs may be focused on business Scenario events are applied to specific businesses or processes. Process lines at a high level or at the process Assessment level. May not have associated root cause Risk management: Explores root cause of loss event. Risk measurement: Root cause analysis. Root Cause not necessarily examined. Management Does not consider impact of reactions. Considers management reaction to catastrophic and extreme but plausible events. Reaction ‘How would we respond?’ ‘What are we exposed to?’ Perspective https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 27/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing 8.9.1 Risk and control self-assessments 8.9.2 Event data There are two main relationships between loss data and scenario analysis. Loss data, comprising internal losses, data consortium-sourced losses and/or publicly-available loss data, is used to illustrate how a scenario may manifest itself. Also a significant loss event may itself be used, in its own right, as the basis for a complete scenario. For example, following the 2008 rogue trading event at Société Générale, the bank’s General Inspection Department published a report, known as the Project Green Report, which was then used by many other firms as the basis for their own scenario analysis covering rogue trading activity. 8.9.3 Risk indicators The relationship between scenario analysis and risk indicators is less direct, given that indicators essentially reflect historical data, while scenarios are trying to analyse future impact. Some firms, having analysed a scenario in detail, then use that information, combined with the causal factors which would be necessary for that scenario to manifest itself, to review their existing risk indicators to determine if any might help detect the scenario before it manifested itself – and if not, what other metrics may have been of use. 8.9.4 Capital measurement Under Basel II banks adopting the “advanced measurement approach” are mandated to use four elements in their capital estimation models. One of the four is scenario analysis data. Similarly, under the European Union’s Solvency II Directive, insurance firms are required to use scenarios as part of internal capital model approvals. Learning activity What other applications for scenario analysis data can you think of, which could deliver meaningful value to a firm? https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 28/29 7/29/24, 10:05 AM Chapter 8 Back Up Book for Printing Summary This chapter has focused on the use of scenario analysis for operational risk management, covering basic concepts relating to scenarios, the objectives and benefits of using scenarios, the different uses of scenarios including challenges, how to identify where scenarios are required, how to document a scenario, factors which affect scenario analysis outcomes and where scenarios fit into the operational risk framework. To aid understanding of this chapter, it is worth reviewing the IOR’s Sound Practice Guidance paper on Scenarios. Key learning Key learning You will be ready to move to the next chapter when you can confidently answer the following questions: 1. What is the purpose of a scenario programme? 2. How many scenarios should a firm use? 3. Why would a firm undertake scenario analysis? 4. What should a firm consider when determining a scenario programme? 5. What approaches could a firm use for scenario analysis? 6. What difficulties may a firm have to overcome to be successful in scenario analysis? 7. What data should a firm collect for a new scenario? 8. What are the types of bias a firm may encounter? 9. How can a firm validate its scenario analysis? 10. What are the differences between RCSA and scenario analysis? 11. What is the relationship between loss data and scenarios? https://www.irmvle.org/mod/book/tool/print/index.php?id=4166&chapterid=2325 29/29