Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models

Martino Ciaperoni, Marzio Di Vece, Roberto Pellungrini, Luca Pappalardo, Fosca Giannotti, Francesco GianniniarXiv 2026 · arXiv

TL;DR

Traditional explainable AI (XAI) fails when language models change behaviors after fine-tuning or prompting because it treats models as static snapshots. This paper introduces Comparative XAI ($XAI_\Delta$), a new framework designed to explain the functional transition between model checkpoints, providing a structured approach to meet emerging AI governance and safety standards.

What Problem Does This Paper Address?

Large Language Models (LLMs) frequently undergo "behavioral shifts"—sudden or systematic changes in safety, accuracy, or alignment—following interventions like fine-tuning, RLHF, or in-context learning.

Existing XAI tools (like feature attribution or probing) are built for single checkpoints. When applied to an updated model, they suffer from explanatory underdetermination: they can tell you what features drive the current output, but not whether those features emerged due to the intervention or were already present in the base model. This lack of causal clarity poses severe compliance risks under new regulations (like the EU AI Act) that mandate documenting causal chains for significant model updates.

The XAI vs XAI_Δ Framework

Methods

The authors formalize the $XAI_\Delta$ paradigm, shifting the target of explanation from a single model's output to the difference in behavior ($\Delta B$) between a reference model ($M_{pre}$) and an updated model ($M_{post}$) given an intervention ($I$).

The framework outlines 10 core desiderata categorized into four pillars:

Figure 2: (a) Taxonomy of XAI∆desiderata organized by category and by whether they pertain to the comparative explainer Φ∆or to the resulting comparative explanation e∆. (b) Bipartite graph mapping the six families of comparative explainers to the desiderata they most naturally support.
  1. Comparability: Ensuring an aligned measurement space (D1), architectural localization (D2), and contrastive relevance (D3).
  2. Validity: Requiring robustness to perturbations (D4) and specificity to the intervention (D5).
  3. Actionability: Demanding causal verifiability via interventions (D6) and operability for mitigation (D7).
  4. Monitoring: Ensuring onset localization over training steps (D8), traceability of artifacts (D9), and clarity of claims (D10).

To demonstrate how to build an $XAI_\Delta$ pipeline, the authors combine multiple techniques: delta-feature attribution (Integrated Gradients), representational similarity (Linear CKA), activation patching, activation steering, Sparse Autoencoders (SAEs), and data attribution.

Datasets

The primary evaluation uses a controlled experimental setup where an instruction-tuned Qwen2.5-0.5B model is fine-tuned via LoRA on a narrow medical dataset, intentionally inducing a harmful behavioral shift (minimizing symptoms and discouraging medical care). Evaluation is conducted on a set of 15 to 50 urgent medical prompts.

Additional validation is performed on Qwen-4B and the inherently interpretable Steerling-8B model using 45 medical prompts under in-context learning conditions.

Key Insights and Ideas

  • The "Snapshot" Illusion: Looking at a fine-tuned model in isolation can mislead engineers. For instance, standard feature attribution might show a model heavily weights the token "headache." However, delta-attribution reveals that the base model already weighted "headache" highly; the true driver of the behavioral shift was a massive increase in the importance of the token "mild" (which led the model to downplay the issue).
Figure 3: Integrated-Gradient attribution differences between Mpost and Mpre reveal shifts in token contributions. Applied to Mpost alone, Integrated Gradients highlights “headache” but cannot separate pre-existing and shift-induced importance; the delta-attribution instead shows that “mild” is the token whose importance increased most after fine-tuning.
  • Causal Levers vs. Correlational Maps: Representation similarity (like CKA) can pinpoint where models diverge in their internal layers, but it doesn't prove that this divergence causes the bad behavior. By combining CKA with activation patching and steering, the framework moves from mere correlation to proving a causal mechanism.

Key Results

  • Localization: Linear CKA successfully isolated the main representational divergence to the third-to-last layer (layer 22) of the Qwen2.5-0.5B model.
  • Causal Verification: Patching hidden states from $M_{pre}$ into $M_{post}$ at layer 22 restored safety scores (evaluated by an LLM-as-a-judge) from a dangerous 2.07/5 back up to 3.67/5 (close to the base model's 4.07/5).
  • Actionable Mitigation: By training a linear probe on layer 22 and subtracting its direction during inference (activation steering with $\alpha=15$), the authors successfully mitigated the harmful shift without modifying any model weights.
  • Concept Shifts: Using SAEs, the authors showed that fine-tuning caused a clear drop in broad "Medical Emergency" concepts and an increase in localized symptom-level concepts.

Limitations and Future Work

The authors note that two of their desiderata were only partially fulfilled in their case study, highlighting areas for future work:

  • Specificity (D5): Ruling out generic training drift requires running placebo fine-tuning on unrelated data to verify that the explanation doesn't appear without a true shift in behavior.
  • Onset Localization (D8): Tracking exactly which training step triggered the shift requires analyzing intermediate checkpoints, which was not performed in this study.
  • Methodological Dependencies: SAE feature labeling remains approximate and highly dependent on the prompt distribution, and using an LLM-as-a-judge introduces model-specific biases.

Summary updated

Abstract

Large-scale foundation models exhibit behavioral shifts when subjected to interventions such as scaling, fine-tuning, reinforcement learning with human feedback, or in-context learning. Current explainability methods are structurally ill-suited to explain these shifts, because they either treat models as static objects, as traditional eXplainable AI (XAI) approaches do, or merely compare independent explanations across different checkpoints of a model. As a result, these approaches fail to explain the functional transition between two model instances in which a certain behavior has shifted following an intervention. This gap creates significant governance risks across jurisdictions including the EU AI Act, US state legislation, and Chinese AI regulations, which require documenting causal chains for substantial system modifications. This position paper argues that explaining behavioral shifts in large language models requires a principled approach that treats the shift itself as the primary object of explanation: namely, one that explains how and why an intervention transforms a reference model into an updated model with different behavior. To support this claim, we introduce Comparative XAI (XAI$_Δ$), a novel XAI paradigm aimed at explaining the difference between two model checkpoints where a behavior has shifted, together with a set of desiderata specifying what XAI$_Δ$ explainers and explanations must satisfy, including comparability, validity, actionability, and monitoring, with the goal of grounding model auditing in explicit, measurable requirements. Finally, we provide preliminary evidence suggesting the need for XAI$_Δ$ in practice through illustrative experiments, compiling the resulting findings into a transition report directly usable for governance and incident documentation.
BibTeX
@misc{ciaperoni2026comparing,
  author = {Martino Ciaperoni and Marzio Di Vece and Roberto Pellungrini and Luca Pappalardo and Fosca Giannotti and Francesco Giannini},
  title = {{Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models}},
  year = {2026},
  eprint = {2602.02304},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.02304},
}
Figure