TL;DR
Traditional explainable AI (XAI) fails when language models change behaviors after fine-tuning or prompting because it treats models as static snapshots. This paper introduces Comparative XAI ($XAI_\Delta$), a new framework designed to explain the functional transition between model checkpoints, providing a structured approach to meet emerging AI governance and safety standards.
What Problem Does This Paper Address?
Large Language Models (LLMs) frequently undergo "behavioral shifts"—sudden or systematic changes in safety, accuracy, or alignment—following interventions like fine-tuning, RLHF, or in-context learning.
Existing XAI tools (like feature attribution or probing) are built for single checkpoints. When applied to an updated model, they suffer from explanatory underdetermination: they can tell you what features drive the current output, but not whether those features emerged due to the intervention or were already present in the base model. This lack of causal clarity poses severe compliance risks under new regulations (like the EU AI Act) that mandate documenting causal chains for significant model updates.
Methods
The authors formalize the $XAI_\Delta$ paradigm, shifting the target of explanation from a single model's output to the difference in behavior ($\Delta B$) between a reference model ($M_{pre}$) and an updated model ($M_{post}$) given an intervention ($I$).
The framework outlines 10 core desiderata categorized into four pillars:
- Comparability: Ensuring an aligned measurement space (D1), architectural localization (D2), and contrastive relevance (D3).
- Validity: Requiring robustness to perturbations (D4) and specificity to the intervention (D5).
- Actionability: Demanding causal verifiability via interventions (D6) and operability for mitigation (D7).
- Monitoring: Ensuring onset localization over training steps (D8), traceability of artifacts (D9), and clarity of claims (D10).
To demonstrate how to build an $XAI_\Delta$ pipeline, the authors combine multiple techniques: delta-feature attribution (Integrated Gradients), representational similarity (Linear CKA), activation patching, activation steering, Sparse Autoencoders (SAEs), and data attribution.
Datasets
The primary evaluation uses a controlled experimental setup where an instruction-tuned Qwen2.5-0.5B model is fine-tuned via LoRA on a narrow medical dataset, intentionally inducing a harmful behavioral shift (minimizing symptoms and discouraging medical care). Evaluation is conducted on a set of 15 to 50 urgent medical prompts.
Additional validation is performed on Qwen-4B and the inherently interpretable Steerling-8B model using 45 medical prompts under in-context learning conditions.
Key Insights and Ideas
- The "Snapshot" Illusion: Looking at a fine-tuned model in isolation can mislead engineers. For instance, standard feature attribution might show a model heavily weights the token "headache." However, delta-attribution reveals that the base model already weighted "headache" highly; the true driver of the behavioral shift was a massive increase in the importance of the token "mild" (which led the model to downplay the issue).
- Causal Levers vs. Correlational Maps: Representation similarity (like CKA) can pinpoint where models diverge in their internal layers, but it doesn't prove that this divergence causes the bad behavior. By combining CKA with activation patching and steering, the framework moves from mere correlation to proving a causal mechanism.
Key Results
- Localization: Linear CKA successfully isolated the main representational divergence to the third-to-last layer (layer 22) of the
Qwen2.5-0.5Bmodel. - Causal Verification: Patching hidden states from $M_{pre}$ into $M_{post}$ at layer 22 restored safety scores (evaluated by an LLM-as-a-judge) from a dangerous 2.07/5 back up to 3.67/5 (close to the base model's 4.07/5).
- Actionable Mitigation: By training a linear probe on layer 22 and subtracting its direction during inference (activation steering with $\alpha=15$), the authors successfully mitigated the harmful shift without modifying any model weights.
- Concept Shifts: Using SAEs, the authors showed that fine-tuning caused a clear drop in broad "Medical Emergency" concepts and an increase in localized symptom-level concepts.
Limitations and Future Work
The authors note that two of their desiderata were only partially fulfilled in their case study, highlighting areas for future work:
- Specificity (D5): Ruling out generic training drift requires running placebo fine-tuning on unrelated data to verify that the explanation doesn't appear without a true shift in behavior.
- Onset Localization (D8): Tracking exactly which training step triggered the shift requires analyzing intermediate checkpoints, which was not performed in this study.
- Methodological Dependencies: SAE feature labeling remains approximate and highly dependent on the prompt distribution, and using an LLM-as-a-judge introduces model-specific biases.
Summary updated