Explainable AI (XAI)

Explainable AI (XAI)

• Addresses the need for machine learning explanations

Why Use Machine Learning?

• Machine learning algorithms may use the wrong reasons for making predictions
• Models could learn X-ray unit types instead of health outcomes when analyzing X-ray images
• Models are vulnerable to adversarial attacks
• Machine Learning Algorithms can be biased
• Algorithms can be easily fooled, even with high confidence predictions for unrecognizable images

Bias in AI

• Amazon scrapped an AI recruiting tool developed that showed bias against women
• The AI penalizes applicants for attending an all-women's college or participating in a women's chess club
• ML models learn decision models based on historical data
• Models can replicate historical biases when making future decisions

Legal Requirements for Explainability

• US Equal Credit Opportunity Act 1974 requires credit agencies to provide the main factors determining credit score
• EU General Data Protection Regulation (GDPR) 2018 gives customers/users the "Right to an explanation" and provides meaningful information about the logic involved in automated decisions

Explainability Defined

• FAT/ML Explainability is defined as ensuring algorithmic decisions and the data driving them can be explained to end-users and other stakeholders in non-technical terms

DARPA's XAI Questions

• Why did you do that?
• Why not something else?
• When do you succeed?
• When do you fail?
• When can I trust you?
• How do I correct an error?

XAI Target Audience

• Domain experts/users of the model: trust the model itself, gain scientific knowledge
• Users affected by model decisions: understand their situation, verify fair decisions
• Regulatory entities/agencies: certify model compliance with legislation, audits
• Data scientists, developers, product owners: ensure/improve product efficiency, research new functionalities
• Managers and executive board members: assess regulatory compliance, understand corporate AI applications

Benefits of ML Explanations

• Validating logic of models
• Defending against adversarial attacks
• Detecting bias
• Regulatory compliance
• Model debugging
• Explainability leads to trust and adoption

Challenges of Explainability

• Model complexity: complex interactions between input variables make it difficult to explain the output as a function of the input
• Decision trees are intrinsically explainable by design
• Interpretable models do not scale
• There is a multiplicity of good models for convex optimization problems

Types of Explainability Options

• White Box Models: self-explanatory; the structure of the model represents the explanation; their outputs are interoperable
• Black Box Models: map user features into a decision class without exposing the how and why they arrive at a particular decision
• Local Explanations: complex models are inherently complex, but a single prediction involves only a small piece of that complexity
• Post-Hoc Explanations: Model Learning > Black Box Model > Surrogate Model Fitting > Interpreted Model > Explanation Generation
• Global
• Local
• Model-Specific
• Model-Agnostic
• Summary of Explainability Options: Explanation Methods > White-Box OR Black Box Prediction Methods

LIME (Local Interpretable Model-Agnostic Explanation)

• ML model approximates an underlying function
• Pros: Widely Cited; Easy to Understand; Easy to Implement
• Cons: Assumes Local Linearity; Computationally Expensive; Requires a large number of samples around explained instance; Not Stable; Approximates the underlying model; Not an exact replica of the underlying model; Fidelity is an open research question
• Conclusion: Explanations can be misleading; Great tool; Very popular, but should be used correctly

SHAP (SHapley Additive exPlanations)

• Shapley Value is a concept in cooperative game theory named after Lloyd Shapley
• Shapley Values are based on game theory for distributing gain in a coalition game
• Shapley Values are a fair way to attribute the total gain to the players based on their contribution
• Credit decisions require several values like: Income, Credit History, No Late Payments, and Number of Credit Products

Shapley Value Axioms

• Dummy: a player never contributes to the game then it must receive zero attribution
• Symmetry: Symmetric players (interchangeable agents) must receive equal attribution
• Efficiency: Attributions must add to the total gain
• Additivity: if model f() is a sum of two other models g() and h(), then the Shapley value calculated for the sum model f() is the sum of Shapley values for model g() and h()

SHAP Explainers

• TreeExplainer: compute SHAP values for trees and ensembles of trees; supports XGBoost, LightGBM, CatBoost and other tree-based models like Random Forest, explainer = shap.TreeExplainer(model)
• DeepExplainer: compute SHAP values for deep learning models by using DeepLIFT and Shapley values; supports TensorFlow and Keras models, explainer = shap.DeepExplainer(model, background)
• GradientExplainer: implementation of expected gradients to approximate SHAP values for deep learning models; supports TensorFlow and Keras models
• KernelExplainer (Kernel SHAP): model agnostics and uses a combination of LIME and Shapley values, explainer = shap.KernelExplainer(model.predict_proba, train_X)

SHAP Visualizations

• Force Plots: Explains a prediction of individual instances
• Dependence Plots
• Summary Plots
• Interaction Values; Pairwise Interaction for TreeExplainer

SHAP Pros & Cons

• Pros: Widely Cited; Based on Theory; Easy to Implement; Comes with a lot of visualization plots
• Cons: expensive to run, however SHAP incorporates LIME into its logic

Study Resources

• "Why should I trust you?" Explaining the predictions of any classifier.
• A Unified Approach to Interpreting Model Predictions.
• Data Camp Article - An Introduction to SHAP Values and Machine Learning Interpretability
• Data Camp – Explainable Artificial Intelligence (XAI)