Explainable AI on the data science interview
Contents:
Why XAI shows up in senior DS loops
Walk into a senior data scientist loop at Stripe, Anthropic, DoorDash, or Airbnb and you will get at least one explainability question before lunch. The hiring manager at Databricks asks "global vs local explanations, when does the distinction matter." The staff scientist at OpenAI hands you a gradient-boosted churn model and asks why customer 1842 was flagged at 0.91 risk. They want the same signal: do you actually understand what your model is doing, or did you ship a black box and pray.
XAI is not academic. The GDPR right to explanation, the Equal Credit Opportunity Act, and the EU AI Act risk classifications all force teams to produce defensible reasons for adverse decisions. A staff DS who cannot articulate the SHAP value for a transaction decline will not pass the loop. The bar is feature attribution that survives a regulator's read.
The less glamorous reason XAI matters: debugging. When your churn model overpredicts on a cohort, SHAP plots are the fastest path to the leaking feature. When your recommender tanks click-through after a launch, LIME on sampled predictions tells you which input drifted. Teams at Linear, Vercel, and Figma keep explanation tooling in their MLOps stack precisely because it shortens incident triage from days to hours.
Global vs local explanations
Interviewers love this distinction because it separates candidates who have shipped models from candidates who have only trained them. A global explanation answers "how does this model behave on average across the input distribution." A local explanation answers "why did this specific input produce this specific output." Both matter. Neither replaces the other.
Global explanations are what you put in a model card, what you show a regulator during an annual audit, and what you discuss with the head of risk at a Tuesday review. Classic outputs are permutation importance bars, partial dependence plots, and mean absolute SHAP per feature. They tell you income drives 28 percent of the variance in your credit model and that referral source is essentially noise. They do not tell you why customer 1842 was denied.
Local explanations are what you hand a customer support agent when a user emails asking why their claim was rejected. They are what you debug with when a fraud analyst flags a false positive. The EU AI Act actually requires them for high-risk systems: per-decision feature attribution a human reviewer can interrogate. When the interviewer at Anthropic asks "would you use SHAP or permutation importance," and the context is "compliance needs to defend a single decision," the answer is local SHAP, not global anything.
A trap that comes up in senior loops: "can you derive a global explanation from a stack of local ones." Yes but with caveats. You can aggregate per-instance SHAP across a representative sample to recover something close to global feature importance. The caveat is that aggregation hides interaction effects and cohort heterogeneity. Saying "just average them" without flagging the caveat signals you have not shipped this at scale.
SHAP from first principles
SHAP stands for SHapley Additive exPlanations, the most asked-about XAI method in 2026 DS loops. Interviewers want two things: a one-sentence intuition, and awareness of which SHAP variant fits which model.
The intuition is from cooperative game theory. Features are players in a coalition producing a prediction. The Shapley value of a feature is its average marginal contribution across every possible ordering of players joining the coalition. The reason it matters: the Shapley value is the unique attribution satisfying four axioms (efficiency, symmetry, dummy, additivity). No other feature attribution method satisfies all four. When the interviewer asks why SHAP and not permutation importance, the answer is the axioms.
import shap
# TreeSHAP for gradient-boosted models
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
# Global summary
shap.summary_plot(shap_values, X_test)
# Local force plot for one prediction
shap.force_plot(
explainer.expected_value,
shap_values[0, :],
X_test.iloc[0, :],
)The variants matter. TreeSHAP runs in polynomial time on tree ensembles and is what you ship with XGBoost, LightGBM, or CatBoost. KernelSHAP is model-agnostic but expensive, so you sample backgrounds and accept Monte Carlo noise. DeepSHAP is built on DeepLIFT for neural networks. GradientSHAP is the integrated-gradients flavor for differentiable models. When the staff scientist at Snowflake asks "you have a 50-million-row XGBoost churn model with sub-100ms latency," the correct answer is TreeSHAP with interventional feature perturbation and a precomputed background sample.
The trap nobody mentions until you fail it: SHAP values are additive in model output space, not probability space. For a logistic classifier they sum to log-odds, not probability. If a junior on your team ships an explanation saying "income contributed +0.34 to your approval probability," they shipped a bug. The fix is either to surface log-odds honestly or to map back through the link function with care, and the senior interviewer wants to hear you say that.
LIME and when it beats SHAP
LIME stands for Local Interpretable Model-agnostic Explanations and predates SHAP by a year. The pitch: to explain a single prediction, fit a sparse linear model on a small perturbed neighborhood around the input, weighted by proximity. The coefficients are your explanation. It works on any model that takes inputs and produces outputs, which is why it remains the default for opaque vendor models where TreeSHAP is not an option.
from lime import lime_tabular
explainer = lime_tabular.LimeTabularExplainer(
X_train.values,
feature_names=X_train.columns,
class_names=["approved", "denied"],
discretize_continuous=True,
)
exp = explainer.explain_instance(
X_test.iloc[0].values,
model.predict_proba,
num_features=10,
)
exp.show_in_notebook()When does LIME beat SHAP. Three scenarios show up in senior interviews. First, when you only have a prediction endpoint and no training data, LIME is your only option. Second, when you need an explanation in milliseconds and TreeSHAP is overkill, LIME on a sampled neighborhood is faster. Third, when stakeholders want a linear narrative ("each $1000 of income shifts approval by 4 percent in this region"), LIME gives you that locally even when the global model is wildly nonlinear.
The catch the interviewer wants to hear: LIME explanations are not stable. Run LIME twice on the same instance and you can get different feature rankings because the sampled neighborhood changes. The fix is to seed the sampler and average multiple runs, but you should bring this up unprompted. Saying "LIME is great, ship it" without acknowledging stability is the moment the staff scientist marks you down.
Counterfactual explanations
Counterfactuals are the explanation type product teams and regulators actually care about. They answer "what is the minimal change to the input that would flip the decision." That is more useful than feature attribution because it is actionable. A customer denied a loan does not benefit from learning their debt-to-income ratio drove 0.4 of the SHAP value. They benefit from learning that paying down $3,200 in revolving debt would have flipped the decision.
import dice_ml
dice_model = dice_ml.Model(model=model, backend="sklearn")
dice_data = dice_ml.Data(
dataframe=df,
continuous_features=["income", "debt", "credit_score"],
outcome_name="approved",
)
exp = dice_ml.Dice(dice_data, dice_model)
cf = exp.generate_counterfactuals(
X_test.iloc[[0]],
total_CFs=3,
desired_class="opposite",
)
cf.visualize_as_dataframe()Good counterfactuals are sparse (change few features), proximal (small changes in feature space), feasible (do not require changing age or country of birth), and diverse (multiple options the user could pursue). DiCE from Microsoft Research handles all four constraints natively. Alibi from Seldon supports counterfactuals with prototypes when you want explanations grounded in actual training-set examples.
The trap: an unconstrained counterfactual will happily suggest impossible changes. Telling a 64-year-old applicant to "become 40" is a textbook example. When you mention counterfactuals in an interview, mention feasibility constraints in the same breath. That separates the candidate who read a blog post from the one who shipped this at Stripe.
Where this shows up in production
Senior interviewers push past methods and ask about the system: how do you serve explanations alongside predictions at scale. Three patterns are worth knowing.
Precomputed explanations: score the model and compute SHAP for every prediction in a nightly batch, write both to a key-value store. Sub-millisecond lookups. Great for daily-batch use cases like credit decisioning. Falls apart for fraud where the input arrives at request time.
Online TreeSHAP: for gradient-boosted models, modern XGBoost or LightGBM bindings expose pred_contribs=True to return SHAP values inline. Explanations cost the same as predictions. This is what Stripe ships for transaction risk. The cost is you can only use tree models.
On-demand sampled: serve predictions in real time, compute explanations only when a customer or analyst requests one, via a downstream service that runs KernelSHAP or LIME on the fly. Latency is fine because the request is human-initiated. Cost is contained because only a small fraction of predictions get explained. Most teams settle on this for product analytics.
Common pitfalls
The first pitfall senior candidates fall into is treating SHAP values as causal. SHAP values are statistical attributions under the model, not causal effects. If your model has learned a spurious correlation between zip code and default risk, SHAP will faithfully attribute risk to zip code, and a regulator will sue you for redlining. The fix is to separate "what is the model doing" from "what is causing the outcome in reality," and to use causal inference tools when the latter is the question on the table.
A related pitfall is conflating global feature importance methods. Permutation importance on a holdout measures how much performance degrades when you scramble a feature. Global SHAP averages local attributions across the training set. Built-in XGBoost importance counts split usage weighted by improvement. These can disagree wildly on the same model. The senior interviewer wants you to name three methods and explain which one you would trust in production, which is permutation importance on a holdout, full stop.
A third pitfall is the additive assumption in SHAP. SHAP decomposes a prediction into per-feature contributions, but the underlying model may rely on strong interactions. SHAP interaction values exist via shap_interaction_values but most teams never look at them because the UX is awful. If the interviewer asks "what about interactions," say yes SHAP supports them, but explanations get exponentially harder for stakeholders to read, and you surface them only when the head of risk specifically asks.
A fourth pitfall is shipping explanations without sanity-checking against training data. The SHAP background drifts away from production input after a few months and explanations start pointing at the wrong features. Refresh the background sample on the same cadence as model retraining, and monitor mean absolute SHAP as a data-quality signal.
A fifth pitfall, the one that fails most candidates, is treating LIME and SHAP as interchangeable. Local SHAP is a globally consistent attribution satisfying additivity. Local LIME is a linear approximation in a sampled neighborhood with no axiomatic guarantees. Change the LIME kernel width, the explanation changes. Change the SHAP background, it changes too but less. Treating them as flavors of the same thing signals you have not used either in anger.
Related reading
- Bayesian methods on the data science interview
- Regression metrics on the data science interview
- Confidence intervals on the data science interview
- Bayesian neural networks interview questions
If you want to drill XAI questions like these every day, NAILDD is launching with hundreds of senior DS problems across exactly this pattern.
FAQ
Is SHAP always the right choice for explanations?
SHAP is the strongest default but not universal. For tree ensembles in production, TreeSHAP wins on cost, axiomatic grounding, and tooling. For deep tabular networks, GradientSHAP or integrated gradients are more faithful and cheaper than KernelSHAP. For opaque vendor models where you only have a prediction endpoint, LIME or sampled KernelSHAP is your only option. The senior signal is naming when each fits, not declaring a universal winner.
How do I explain a model to a non-technical stakeholder?
Skip the SHAP plot. Lead with a counterfactual. "The customer was denied because their debt-to-income ratio was 0.62. If it had been 0.48, the model would have approved." Then show the top three drivers as plain English sentences with magnitudes in business units. A non-technical stakeholder cannot read a force plot but can read a sentence, and a regulator accepts the counterfactual narrative more readily than a bar chart.
Does explainability hurt model performance?
Not directly. Adding SHAP after training does not change the model. The tradeoff shows up in model choice. If compliance mandates explainability, you may be pushed toward gradient-boosted trees over deep tabular networks even when the latter would win on AUC. That is a constraint, not a cost of XAI itself. The constraint forces you to ship a model the business can actually defend, which is worth a few points of AUC most of the time.
When should I use counterfactuals instead of SHAP?
Use counterfactuals when the audience needs to act on the explanation. A denied loan applicant wants to know what to change. A fraud analyst wants to know what would have cleared the transaction. Use SHAP when the audience needs to understand the model's logic without acting on it: model cards, regulator audits, debugging sessions.
How does the EU AI Act change explainability requirements?
For high-risk systems (credit scoring, employment, biometric ID, critical infrastructure, education access), the EU AI Act requires technical documentation: model logic, performance metrics on protected groups, and a process for human oversight of individual decisions. In practice that means model cards plus per-decision explanations a human reviewer can interrogate. SHAP plus counterfactuals are the de facto standard. Senior DS candidates are expected to know the broad strokes.