NPEFF: Non-Negative Per-Example Fisher Factorization

Michael Matena, Colin RaffelarXiv 2023 · DOI · arXiv

Introduction

Interpretability research has largely focused on activation space, using methods like Sparse Autoencoders (SAEs) to identify "monosemantic" features—internal directions representing specific concepts like "legal text" or "dogs." However, identifying what features a model detects does not fully explain its processing strategies—the heuristics used to transform those features into predictions.

This paper introduces Non-Negative Per-Example Fisher Factorization (NPEFF), a novel method that shifts the focus to parameter space. By decomposing Fisher information matrices, NPEFF uncovers the functional substeps or "computational strategies" models employ, such as specific routines for handling dates or identifying sentiment.

Methodology

Analyzing parameter-level changes in large language models (LLMs) is traditionally computationally prohibitive due to the size of the Fisher information matrices ($N \times N$ for $N$ parameters). The authors overcome this through a multi-step pipeline:

  1. Low-Rank Approximations: Using sparse random projections and Singular Value Decomposition (SVD), they create tractable representations of per-example Fisher (PEF) matrices.
  2. NPEFF Decomposition: A specialized non-negative factorization algorithm decomposes these PEFs into rank-1 positive semi-definite (PSD) matrices. Each component represents a distinct model strategy.
  3. Selective Perturbation: Using compressed sensing to reverse the projections, researchers can nudge model weights in the direction of a specific component to selectively disrupt or enhance a strategy without affecting overall performance.

Datasets and Strategy Identification

The researchers evaluated NPEFF across BERT-base and SmolLM2 models using three primary benchmarks: SNLI (logical entailment), SST2 (sentiment analysis), and TriviaQA (open-domain QA).

Examples of identified computational strategies across SNLI, SST2, and TriviaQA, showing how the model groups functionally similar inputs.

As shown in Figure 2, NPEFF identifies highly specific themes. For instance, in TriviaQA, it might isolate a strategy specifically for "years of death," while in SNLI, it identifies how the model processes contradictions regarding colors or clothing.

Key Insights

  • Strategies vs. Features: Unlike SAE features (which might detect a "year"), NPEFF identifies the logic used to process that year.
  • Hierarchical Splitting: As the decomposition granularity increases, general strategies (e.g., "sentiment") split into more specific ones (e.g., "negative sentiment").
  • Generalization: Strategies discovered on training data remain consistent on held-out data, suggesting they represent fundamental model behaviors rather than dataset artifacts.
  • In-Context Learning (ICL): NPEFF reveals that ICL often works by helping the model select and adapt existing internal strategies rather than learning entirely new ones from the prompt.

Quantitative Results

The effectiveness of NPEFF was validated through both human evaluation and functional testing:

  • Interpretability: In human trials for TriviaQA, 79% of components were found to have a consistent, detectable theme.
  • Consistency: On the SNLI dataset, 92.6% of components maintained the same tuning (prediction/label alignment) across training and held-out sets.
  • Selectivity: The most rigorous proof of the method's validity is its selectivity. By perturbing the model in the direction of a specific Fisher vector, the authors could increase the KL-divergence (a measure of output change) on targeted examples by 10-18x compared to random examples.
Selectivity results demonstrating the functional link between NPEFF components and specific model behaviors via KL-divergence ratios.

As detailed in Figure 6, the KL-ratio for BERT-BASE-NLI reached 18.59, confirming that these components are functionally linked to specific, isolatable model behaviors.

Conclusion

NPEFF provides a necessary bridge between feature detection and behavioral analysis. While computational intensity remains a challenge for the largest frontier models, the ability to map activation-space features to parameter-space strategies offers a path toward building complete "circuits" of model reasoning. This work has significant implications for targeted model editing, unlearning, and safety alignment.

Summary updated

Abstract

We introduce NPEFF (Non-Negative Per-Example Fisher Factorization), an interpretability method that aims to uncover strategies used by a model to generate its predictions. NPEFF decomposes per-example Fisher matrices using a novel decomposition algorithm that learns a set of components represented by learned rank-1 positive semi-definite matrices. Through a combination of human evaluation and automated analysis, we demonstrate that these NPEFF components correspond to model processing strategies for a variety of language models and text processing tasks. We further show how to construct parameter perturbations from NPEFF components to selectively disrupt a given component's role in the model's processing. Along with conducting extensive ablation studies, we include experiments to show how NPEFF can be used to analyze and mitigate collateral effects of unlearning and use NPEFF to study in-context learning. Furthermore, we demonstrate the advantages of NPEFF over baselines such as gradient clustering and using sparse autoencoders for dictionary learning over model activations. We release the code used in this work.
BibTeX
@misc{matena2023npeff,
  author = {Michael Matena and Colin Raffel},
  title = {{NPEFF: Non-Negative Per-Example Fisher Factorization}},
  year = {2023},
  eprint = {2310.04649},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.04649},
}
Figure