Appearance
The Reliability Problem in Interpretability: Four Papers, Four Angles
Model auditing has a measurement problem. You can measure accuracy, latency, calibration. You can't yet measure whether an explanation is true.
Four recent arXiv preprints tackle that gap head-on. They come from different corners of the field: representation geometry, EEG decoding, formal verification, and human-computer interaction. Read together, they tell one story. Every explanation is a claim with a validity condition attached, and most current tooling skips the condition.
The field can now extract features from transformer layers, attribute decisions to input channels, and render explanations in natural language. Each of those capabilities just got a paper that tests its limits. The results are uncomfortable, and they're exactly what you need if you're the person who has to sign off on a model's behavior.
Transformer updates preserve direction more than we assumed
The first paper, Disentangling Representation Evolution in Transformers through Directional Decomposition (arXiv:2609.15975), looks at what a transformer layer actually does to the hidden state.
Each layer computes h_{l+1} = h_l + Δ. The common mental model says the residual stream preserves direction, and each layer just adds new content. The paper decomposes Δ into a component parallel to h_l (it scales the existing direction) and a component perpendicular to it (it redirects). Across pretrained models, the parallel component is substantial beyond the identity path. Layers push along existing directions more than people assumed.
Why this matters for auditing: an explanation that says "this layer amplifies feature X" is a claim about parallel manipulation. An explanation that says "this layer routes information into new directions" is a claim about perpendicular manipulation. They have different stability profiles, and the paper shows it.
Edits live or die by their space
Apply the decomposition in two spaces and the behavior diverges. Residual space: attention and MLP updates relative to the hidden state. Value space: attention value aggregation relative to the current token's value.
Targeted edits reveal a strong asymmetry. Exclude-self value-space parallel manipulation is markedly more robust than residual-space or perpendicular counterparts. If you want to scale everything a token absorbs from other tokens while keeping its direct self message intact, do it in value space. The same edit in residual space breaks down.
The decomposition also works as a diagnostic. When models are compressed (quantization, low-rank approximations), update error splits into parallel and perpendicular parts. Perpendicular error separates compression methods more clearly than parallel error. If you're comparing compression approaches for an audited model, measure the perpendicular component of the update error first.
There's a training-time payoff too. Suppressing full-aggregate parallel updates during from-scratch pretraining lowers validation-loss trajectories and improves downstream averages. The value-space variant produces the strongest gains. Constraining update geometry during pretraining improves the loss trajectory directly.
Quick Take: Explanation robustness depends on which representational space an edit lives in. Parallel manipulation in value space survives; the same intervention in residual space does not.
EEG foundation models need their own explainability
EEG foundation models like BIOT, LaBraM, and EEGMamba decode neural signals with strong benchmark numbers. But a black-box seizure detector is useless to a neurologist who has to justify a treatment decision. EEG-Xplain (arXiv:2609.15687) builds a unified attribution framework across those heterogeneous architectures.
The pipeline runs gradient-, perturbation-, and activation-based methods side by side, then projects results into three clinical dimensions:
- Spatial: which EEG channels matter, rendered as topographic maps.
- Temporal: which signal segments drive the decision.
- Frequency: how canonical rhythms (delta, theta, alpha, beta, gamma) contribute, via spectral perturbation.
Validation is population-level, not anecdotal. EEG-Xplain combines Area Over the Perturbation Curve (AOPC) with cross-method consistency: an explanation isn't trusted unless perturbation-based removal of the attributed signal actually degrades the prediction, and unless different attribution families agree on where the signal is.
On Mumtaz2016 and TUAB, the explanations match established neurophysiological markers. That's the good news. The paper also exposes dependencies on artifacts and spurious patterns. In a clinical setting, an attribution framework that catches "the model is reading the electrode noise" is worth more than one that only produces pretty maps.
The last piece is an LLM that converts structured attribution outputs into natural-language reports. That closes the loop between scalp voltages and a sentence a clinician can skim.
Key numbers
- Five model families exhibit IRN feature flips under minor perturbations: GPT-2 small, Gemma 2 2B, Gemma 3 1B, Llama 3.2 1B, R1-Distill-Qwen 1.5B. Every one fits on a consumer GPU, so you can verify this locally.
- Two decomposition spaces drive editing robustness: residual space (updates relative to the hidden state) and value space (aggregation relative to the current token).
- 76.4% and 74.3% of non-expert respondents rated What-if and Why-not narrative explanations as more comprehensible than traditional SHAP plots.
Interpretable replacement networks are fragile
The third paper is bluntly titled The Misery of Mechanistic Interpretability (arXiv:2609.15533). Its target is interpretable replacement networks (IRNs): sparse, interpretable modules trained to replace the layers of a larger model, exposing human-readable features.
Standard IRN evaluation checks faithfulness on clean data. The paper shows that semantically minor input perturbations flip the dominant features. GPT-2 small (124M parameters, runs on a laptop), Gemma 2 2B, Gemma 3 1B, Llama 3.2 1B, and R1-Distill-Qwen 1.5B all fail the same way. This is not a frontier-scale pathology. Small models carry the same disease.
When I walked through the experiment setup, the failure mode is easy to reproduce mentally: substitute a few tokens, keep the sentence grammatical and the meaning intact, and the base model's prediction barely moves while the replacement network's dominant feature flips to something semantically unrelated. The interpretation an auditor would read changes completely. The model didn't change at all.
An IRN explains a specific input rather than the model.
The four papers attack different layers of the same problem:
| Paper | Attack surface | Method | Headline finding |
|---|---|---|---|
| Directional decomposition (2609.15975) | Transformer representation geometry | Parallel/perpendicular decomposition of layer updates | Exclude-self value-space edits are robust; perpendicular error separates compression methods best |
| EEG-Xplain (2609.15687) | EEG foundation models | Multi-method attribution, AOPC validation, LLM reports | Explanations match neurophysiology but expose artifact dependence |
| Misery of mechanistic interpretability (2609.15533) | Interpretable replacement networks | Formal reachability verification | Minor perturbations flip dominant features across five model families |
| Data storytelling + IML (2609.15722) | Non-expert end users | What-if/Why-not narrative generation over SHAP | Most respondents prefer stories over raw SHAP visualizations |
Formal verification puts a number on faithfulness
The misery paper's negative result would be depressing if it stopped there. It doesn't. It introduces the first formal verification framework for IRN faithfulness, based on reachability analysis. The verifier computes a sound upper bound on the faithfulness gap under adversarial perturbations.
A certified bound gives an auditor something concrete: under any input in a defined region, the interpretation error stays below a proven threshold. That's a statement you can write into a review, unlike an empirical accuracy number on a held-out set.
The second half of the result is that verification-aware training substantially tightens the certified bound. Train the IRN with the reachability bound in the loss, and the model learns to keep its interpretations stable inside the certified region. The paper reports this restores a feature-level interpretation that safety auditors can rely on, with a formal guarantee attached.
Scope check: the experiments cover small open-weight models, and the bounds are conservative by design. Formal verification of interpretability at frontier scale is not here yet. But the framework is the first credible answer to "how do you know the explanation is true?"
Stories beat charts for the people who read the report
The last paper (arXiv:2609.15722) starts from a different failure: even a correct explanation fails if the audience can't parse it. SHAP visualizations assume a reader who understands Shapley values. Most compliance reviewers don't.
The contribution is a design framework rather than a single model: a formal definition of data storytelling in interpretable ML, the DIST Pyramid that aligns storytelling with explanation goals, and an I-P-O model describing their interaction. The implementation generates What-if and Why-not explanations through separate event-generation processes, applies data desensitization so sensitive inputs never appear in the story, and renders the result through LLMs using an And-But-Therefore (ABT) narrative structure.
The case study uses Boston Housing with SHAP values powering the generated stories. The empirical result: 76.4% of respondents rated the What-if data stories as more comprehensible than traditional SHAP visualizations, and 74.3% said the same for the Why-not stories. Accessibility scores were significantly higher for both.
Three in four non-experts understood the story version better. If your interpretability output is consumed by people outside the ML team, SHAP plots are the wrong deliverable.
Common pitfalls
Trusting a single attribution method. Gradient, perturbation, and activation attributions disagree on real inputs. EEG-Xplain runs all three and checks consistency precisely because single-method explanations are the easiest to fool. Pair at least one perturbation-based method with a gradient-based one.
Evaluating replacement networks on clean data only. Clean-data faithfulness is the default in most interpretability papers, and it's the exact condition under which IRN dominant features flip. Add a perturbation budget to any IRN evaluation and report the flip rate.
Editing in the wrong representational space. The directional decomposition shows that an identical intervention in residual space and value space has different robustness profiles. If you want to scale non-self token aggregation, edit value space with exclude-self masking. Reaching for residual-space edits first is a common reflex that produces brittle results.
Showing SHAP values to stakeholders. SHAP is a debugging tool, not a communication tool. The Boston Housing study shows three quarters of non-experts do better with a story. Desensitize the data, generate What-if/Why-not narratives, keep the SHAP in the appendix.
Confusing a certified bound with a complete guarantee. Reachability bounds are sound but conservative, and the experiments cover small open-weight models. A certified faithfulness gap under adversarial input is a major improvement, but it's not a license to skip behavioral testing.
One thing to remember
All four papers converge on the same lesson from different directions: an explanation is a claim about a model, and every claim needs a validity condition. The geometry paper specifies where edits hold (value space, parallel). The EEG paper specifies how explanations are validated (AOPC, consistency, neurophysiological markers). The reachability paper specifies the input region where faithfulness is bounded. The storytelling paper specifies the audience who can actually use the explanation. State the validity condition, or the explanation isn't done.
The Bottom Line
- If you're auditing transformer interpretability claims, expect robust findings to be value-space parallel manipulations. Treat residual-space explanations as fragile until proven otherwise.
- If you're deploying EEG foundation models in clinical settings, require multi-method attribution with AOPC and consistency checks from day one. Single-method attributions on BIOT or LaBraM score well on benchmarks but still read electrode artifacts.
- If your safety story relies on interpretable replacement networks, perturb the input before trusting any feature-level interpretation. Verification-aware training is moving fast, so plan for certified faithfulness bounds to become standard in audit workflows within a year.