Introduction
Interpretability research has largely focused on activation space, using methods like Sparse Autoencoders (SAEs) to identify "monosemantic" features—internal directions representing specific concepts like "legal text" or "dogs." However, identifying what features a model detects does not fully explain its processing strategies—the heuristics used to transform those features into predictions.
This paper introduces Non-Negative Per-Example Fisher Factorization (NPEFF), a novel method that shifts the focus to parameter space. By decomposing Fisher information matrices, NPEFF uncovers the functional substeps or "computational strategies" models employ, such as specific routines for handling dates or identifying sentiment.
Methodology
Analyzing parameter-level changes in large language models (LLMs) is traditionally computationally prohibitive due to the size of the Fisher information matrices ($N \times N$ for $N$ parameters). The authors overcome this through a multi-step pipeline:
- Low-Rank Approximations: Using sparse random projections and Singular Value Decomposition (SVD), they create tractable representations of per-example Fisher (PEF) matrices.
- NPEFF Decomposition: A specialized non-negative factorization algorithm decomposes these PEFs into rank-1 positive semi-definite (PSD) matrices. Each component represents a distinct model strategy.
- Selective Perturbation: Using compressed sensing to reverse the projections, researchers can nudge model weights in the direction of a specific component to selectively disrupt or enhance a strategy without affecting overall performance.
Datasets and Strategy Identification
The researchers evaluated NPEFF across BERT-base and SmolLM2 models using three primary benchmarks: SNLI (logical entailment), SST2 (sentiment analysis), and TriviaQA (open-domain QA).
As shown in Figure 2, NPEFF identifies highly specific themes. For instance, in TriviaQA, it might isolate a strategy specifically for "years of death," while in SNLI, it identifies how the model processes contradictions regarding colors or clothing.
Key Insights
- Strategies vs. Features: Unlike SAE features (which might detect a "year"), NPEFF identifies the logic used to process that year.
- Hierarchical Splitting: As the decomposition granularity increases, general strategies (e.g., "sentiment") split into more specific ones (e.g., "negative sentiment").
- Generalization: Strategies discovered on training data remain consistent on held-out data, suggesting they represent fundamental model behaviors rather than dataset artifacts.
- In-Context Learning (ICL): NPEFF reveals that ICL often works by helping the model select and adapt existing internal strategies rather than learning entirely new ones from the prompt.
Quantitative Results
The effectiveness of NPEFF was validated through both human evaluation and functional testing:
- Interpretability: In human trials for TriviaQA, 79% of components were found to have a consistent, detectable theme.
- Consistency: On the SNLI dataset, 92.6% of components maintained the same tuning (prediction/label alignment) across training and held-out sets.
- Selectivity: The most rigorous proof of the method's validity is its selectivity. By perturbing the model in the direction of a specific Fisher vector, the authors could increase the KL-divergence (a measure of output change) on targeted examples by 10-18x compared to random examples.
As detailed in Figure 6, the KL-ratio for BERT-BASE-NLI reached 18.59, confirming that these components are functionally linked to specific, isolatable model behaviors.
Conclusion
NPEFF provides a necessary bridge between feature detection and behavioral analysis. While computational intensity remains a challenge for the largest frontier models, the ability to map activation-space features to parameter-space strategies offers a path toward building complete "circuits" of model reasoning. This work has significant implications for targeted model editing, unlearning, and safety alignment.
Summary updated