Permutation importance in DS interviews

Why interviewers love this question

You are interviewing for a Data Scientist role at Stripe. The hiring manager pulls up a notebook with an XGBoost model trained on payment fraud data, points at model.feature_importances_, and asks: "Why don't you trust this output? What would you compute instead?" The expected answer is permutation importance — and the reason it shows up in 1 out of 3 mid-level DS loops is that it forces you to reason about what "important" actually means when a feature is correlated, leaked, or just memorized.

Permutation importance is also one of those rare topics where the right answer is short, the wrong answer is short, and the difference between them is exactly the gap between a candidate who has shipped a model and one who has only read about gradient boosting. If you can explain the algorithm in 60 seconds and name two failure modes, you pass this question.

The other reason it shows up so often: it is model-agnostic. You can compute it for a random forest, a neural net, or a logistic regression with the same three lines of scikit-learn. That makes it the default interpretability tool when the team rotates models every quarter, and it makes it a fair question regardless of which framework you put on your resume.

The algorithm

The core idea is simple: if a feature matters, shuffling its values should destroy the model's score. If a feature is noise, shuffling it should change nothing. You repeat this for every feature, average over a few shuffles to dampen randomness, and rank.

1. Train the model on the training set.
2. Compute a baseline metric M0 on a held-out set (AUC, RMSE, log-loss).
3. For each feature f:
   a. Shuffle the values of f across rows (break the link to the target).
   b. Predict with the model — weights stay frozen.
   c. Compute the new metric M_f.
   d. Importance(f) = M0 - M_f   (drop in performance).
   e. Restore the original column.
4. Repeat steps 3a-3d n_repeats times and average.

In scikit-learn:

from sklearn.inspection import permutation_importance

result = permutation_importance(
    model, X_val, y_val,
    scoring="roc_auc",
    n_repeats=10,
    random_state=42,
    n_jobs=-1,
)

for i in result.importances_mean.argsort()[::-1]:
    print(f"{X_val.columns[i]:30s} "
          f"{result.importances_mean[i]:.4f} "
          f"+/- {result.importances_std[i]:.4f}")

Two things to notice. First, model weights never change — you only re-score, which makes this much cheaper than retraining. Second, the output has both a mean and a standard deviation across repeats. Always report the standard deviation — if it is the same order of magnitude as the mean, the ranking is noise.

Permutation vs feature_importances_

This is the comparison interviewers want to hear, almost word for word. The shortest defensible answer:

The most damning example in interviews is the leaky timestamp case. An XGBoost model on a churn task shows signup_timestamp as importance 0.42. Permutation importance on the validation set returns 0.001 +/- 0.003 — statistically indistinguishable from zero. The model was using the timestamp to memorize training cohorts, and feature_importances_ happily rewarded that. Permutation caught it because validation rows didn't reward memorization.

Train, validation, or test set?

The honest answer is it depends on the question, and saying so is worth more points than picking one and defending it religiously.

Run permutation on the training set when you want to know which features the model is actually using — useful for debugging, for spotting features the model decided to ignore, and for diagnosing whether regularization is too aggressive. A feature with high train-set permutation importance but low validation-set importance is a memorization signal.

Run permutation on the validation or test set when you want to know which features generalize — this is the production-relevant answer and the one you should default to unless the interviewer specifies otherwise. The interview-safe phrasing: "Validation by default, train when I'm diagnosing overfitting."

Common pitfalls

The trap that fails the most candidates is correlated features. If revenue_30d and revenue_60d carry overlapping signal, permuting one of them barely hurts the model — the other still provides the same information. Both columns get low importance scores even though together they are critical. The textbook fix is group permutation: shuffle the entire correlated cluster at once and assign the importance to the group. In practice, run a clustering step on the feature correlation matrix (sklearn's hierarchy.fcluster over Spearman distance is the standard recipe) and permute clusters, not columns.

The second pitfall is interpreting marginal importance as causal. Permutation importance answers "how much does the model rely on this feature", not "how much does this feature cause the outcome". A user who confuses the two will recommend dropping a feature with low permutation importance and watch their model break — because the feature was carrying a weak signal that mattered in a specific cohort. The fix is to treat permutation importance as a screening tool, not a feature-selection oracle. Confirm with retraining without the feature before you delete it.

The third pitfall is using accuracy on imbalanced data. If your positive class is 2%, a model can drop almost any feature and still score 98% accuracy. Permutation importance computed against accuracy will return mostly zeros and miss real signal. Switch to AUC, average precision, or log-loss — metrics that move when probability calibration shifts, not only when the argmax flips.

The fourth pitfall is runtime on wide datasets. The cost scales as n_features × n_repeats × prediction_cost. On a dataset with 800 features, 10 repeats, and an XGBoost model that takes 4 seconds to predict on the validation set, you are looking at roughly 9 hours of compute. Two fixes work in practice: subsample the validation set (10k rows is usually enough to stabilize the top 20), and parallelize with n_jobs=-1. If you have over 1,000 features, you almost certainly want SHAP — a single forward pass gives you per-row attributions and is cheaper for trees.

The fifth pitfall, and the one that catches senior candidates, is using a single repeat. With n_repeats=1 the importance estimate is one draw from a noisy distribution. Two features with true importance of 0.01 and 0.012 will swap rank half the time. The default n_repeats=5 is fine for screening; bump to 10-30 when you are about to make a decision based on the ranking.

Practical tips

Cache the baseline metric outside the loop — scikit-learn does this for you, but rolling your own implementation is a common interview follow-up, and forgetting the cache is an easy way to double the runtime.

Use stratified shuffling if your target is heavily imbalanced. The default np.random.permutation ignores the target distribution, which is fine for AUC but can produce odd results for class-conditional metrics. Most teams skip this in practice and accept the small bias.

When reporting importances to stakeholders, plot mean +/- 1 std as horizontal bars sorted descending, and draw a vertical line at zero. Features whose error bars cross zero are not significant, and you should say so out loud. This single chart prevents 80% of the "but why did you drop my favorite feature" arguments.

For tree models specifically, prefer SHAP for production explanations and permutation importance for the audit. SHAP is faster on trees (TreeSHAP runs in polynomial time), gives per-row attributions, and respects feature interactions. Permutation importance is still the right tool for the global ranking and for any non-tree model.

Related reading

• Feature engineering — DS interview
• Decision trees — DS interview
• Bagging vs Boosting — DS interview
• XGBoost vs LightGBM vs CatBoost — DS interview
• Bias-variance tradeoff — DS interview
• Explainable AI — DS interview

If you want to drill DS interview questions like this every day, NAILDD ships with 1,500+ problems across stats, ML, and SQL — including a full track on model interpretability.

FAQ

Is permutation importance always better than feature_importances_?

For ranking generalizable features on a tree model, yes — permutation importance computed on validation data is the more honest signal. But feature_importances_ is free, runs at fit time, and is good enough for rough debugging on small models. The right rule is: use feature_importances_ for first-look exploration and permutation importance whenever the decision matters (feature selection, presenting to stakeholders, audits). On non-tree models, permutation importance is the only option from this pair.

Why does my permutation importance report negative numbers?

A negative value means shuffling the feature improved the metric, which usually indicates one of three things: the feature was noise and the model was slightly overfitting to it, the metric is unstable on a small validation set, or the feature actively hurts predictions on this slice. Treat anything within +/- std of zero as effectively zero, and investigate large negatives — they often flag data leakage from the past into the future after a quick join error.

How many repeats should I use?

For exploration, n_repeats=5 (the scikit-learn default) is fine. For any decision — dropping a feature, presenting to leadership, writing a model card — push it to 10-30. The standard deviation across repeats is your trust signal: when it shrinks below 20% of the mean, you have a stable ranking. On expensive models, subsample the validation set rather than cutting repeats.

Does permutation importance work for time-series models?

It works, but with a caveat: shuffling a feature across time rows breaks the temporal structure. If your model uses a lag feature like revenue_t-1, naive permutation will both shuffle the values and break the relationship with revenue_t-2. The robust pattern is block permutation — shuffle blocks of consecutive rows so the local time structure survives. For most production forecasting work, SHAP with a time-aware reference distribution is a cleaner answer.

Permutation importance vs SHAP — which one for an interview?

If the interviewer asks for a global feature ranking and you want to demonstrate understanding of generalization, lead with permutation importance. If they ask "why did the model predict X for this user" or about local explanations, lead with SHAP. The senior-level answer mentions both and chooses based on the use case: permutation for audits, SHAP for per-prediction explanations and for production debugging on trees.

Can I use permutation importance for feature selection?

You can, but it is not the most efficient tool. The recipe is: compute permutation importance on validation, drop features whose mean is within one standard deviation of zero, retrain, and verify the validation metric did not drop. For large feature sets, this is slow — you are paying the permutation cost on every iteration. Recursive feature elimination with a cheap estimator, or L1-regularized linear models, will get you a candidate set faster. Permutation importance is best as the final sanity check before you commit to the reduced feature set.