Explainable AI (XAI)

Explainable AI (XAI)

• XAI is the field of Machine Learning dedicated to making Machine Learning models understandable and interpretable

The Need for Machine Learning Explanations

• Models can be right for the wrong reason, such as health outcome predictions being based on X-ray unit type instead of the image itself
• Machine learning algorithms are easily fooled by adversarial attacks, which are small perturbations to the input that cause the model to make incorrect predictions
• Machine learning algorithms can be biased, leading to unfair or discriminatory outcomes
• Amazon scrapped an AI recruiting tool that showed bias against women

Explainability as a legal requirement

• The US Equal Credit Opportunity Act of 1974 requires credit agencies to provide the main factors determining credit score
• EU General Data Protection Regulation (GDPR) 2018 includes a "right to an explanation," providing affected customers/users 'meaningful information about the logic involved' in automated decisions

Explainability defined

• Ensure that algorithmic decisions, as well as any data driving those decisions, can be explained to end-users and other stakeholders in non-technical terms.

DARPA's Explainable Artificial Intelligence (XAI) program aims to address the following questions

• Why did you do that?
• Why not something else?
• When do you succeed?
• When do you fail?
• When can I trust you?
• How do I correct an error?

XAI Target Audience

• Domain experts/users of the model need to trust the model itself and gain scientific knowledge
• Those affected by model decisions need to understand their situation and verify fair decisions
• Regulatory entities/agencies need to certify model compliance with the legislation in force and audits
• Data scientists, developers, and product owners need to ensure/improve product efficiency, research, and new functionalities
• Managers and executive board members need to assess regulatory compliance and understand corporate AI applications

Benefits of Machine Learning Explanations

• Validating the logic of models
• Defending against adversarial attacks
• Detecting bias
• Regulatory compliance
• Model debugging

Model Complexity Makes Explainability Difficult

• Machine learning methods achieve better accuracy through learning complex functions
• These functions intrinsically establish multiple interactions between input variables, making it difficult to explain the output as a function of the input

Decision Trees

• Decision trees are intrinsically explainable by design

Model Scale

• Interpretable models' explanations don't scale

Using Interpretable Models

• Use interpretable models if you can

GPT-3 Size

• OpenAI’s Natural Language Processing Model has 175,000,000,000 parameters

Multiplicity of Good Models

• The multiplicity of good models makes interpretability difficult for convex optimization problems

White Box Models

• White Box Models are self-explanatory and the structure of the model represents the explanation
• White Box Models have interoperable output

Black Box Models

• Black-Box Models map user features into a decision class without exposing the how and why they arrive at a particular decision

Local Explanation

• Complex models are inherently complex, but a single prediction involves only a small piece of that complexity

Post-Hoc Explanations

• Post-Hoc Explanation consists of first establishing a model, then having a surrogate model fitted to the black-box model, and finally explaining the surrogate model

Summary of Explainability Options

• White-Box are intrinsic and explainable
• Black Box Prediction Methods can be Global, Local, Model-Specific, Model-Agnostic

LIME

• LIME: Local Interpretable Model-Agnostic Explanations is a technique where the ML model approximates an underlying function

How LIME works

• Black Box model takes an input and produces an output
• LIME is used to create an explanation from the Black Box output
• A local linear model is constructed around a data instance of interest to approximate the behavior of the black box model in that local region.
• This simplified model provides insights into the feature contributions for the specific prediction

LIME pros

• Widely cited
• Easy to understand
• Easy to implement

LIME cons

• Assumes local linearity
• Computationally expensive
• Requires a large number of samples around the explained instance
• Not stable
• Approximates the underlying model
• It is not an exact replica of the underlying model
• Fidelity is an open research question

LIME conclusion

• A misleading explanation can be used to fool users into trusting a biased classifier
• It is a great and very popular tool
• Use LIME carefully and do not blindly use it!

SHAP

• SHAP (SHapley Additive exPlanations): Is a technique used to explain predictions based off of Shapley Values

Shapley Value

• Shapley Value is a concept in cooperative game theory named after Lloyd Shapley
• Members should receive payments or shares proportional to their marginal contributions
• Lloyd Shapley introduced his theory in 1951
• Lloyd Shapley won the Nobel Prize in Economics in 2012

Shapley Values

• Shapley Values based on game theory for distributing gain in a coalition game
• Players in the game collaborate to generate some gain (value)
• Shapley Values are a fair way to attribute the total gain to the players based on their contribution

Explanation using Shapley Values

• Explaining the Model Output is like a Coalition Game
• Prediction corresponds to Gain/Payout
• Input Features corresponds to Players

Shapley Value Axioms

• Dummy: If a player never contributes to the game then it must receive zero attribution
• Symmetry: Symmetric players (interchangeable agents) must receive equal attribution
• Efficiency: Attributions must add to the total gain
• Additivity: if model f() is a sum of two other models g() and h(), then the Shapley value calculated for the sum model f() is the sum of Shapley values for model g() and h()

SHAP's Additive Feature Attribution

• Important data to decide whether to provide credit is usually income, credit history, number of late payments and the number of credit products already owned by the applicant
• SHAP aims to identify the average contribution of income to this decision, given different combinations of other features

SHAP Explainers Include

• TreeExplainer: computes SHAP values for trees and ensembles of trees
  • Supports XGBoost, LightGBM, CatBoost, and other tree-based models like Random Forest
• DeepExplainer: computes SHAP values for deep learning models by using DeepLIFT and Shapley values
  • Supports TensorFlow and Keras models
• GradientExplainer: an implementation of expected gradients to approximate SHAP values for deep learning models
  • Supports TensorFlow and Keras models
• KernelExplainer (Kernel SHAP): model agnostics and uses a combination of LIME and Shapley values

SHAP Visualization plots

• Force Plots: Used to explain the prediction of individual instances
• Dependence Plots
• Summary Plots
• Interaction Values

SHAP Pros

• Widely Cited
• Based on Theory
• Easy to Implement
• Comes with a lot of visualization plots

SHAP Conclusion

• Great tool and very popular
• Based on game theory
• Supports many useful visualization charts
• Expensive to run
• Don’t think of it as an alternative to LIME because Kernel SHAP incorporates LIME into its logic