Accountability Attribution: Tracing Model Behavior to Training Processes

Shichang Zhang, Hongzhe Du, Karim Saraipour, Jiaqi W. Ma, Himabindu Lakkaraju

TL;DR

Modern foundation models undergo complex multi-stage pipelines (pretraining, fine-tuning, alignment), making it difficult to pinpoint which training phase caused a specific behavior. This paper introduces accountability attribution, a causal framework based on potential outcomes and first-order Taylor approximations that calculates the impact of any training stage without expensive retraining. By factoring in realistic optimization dynamics—such as learning rate schedules, momentum, and weight decay—the method reliably pinpoints stages responsible for both beneficial capabilities and harmful shortcuts.

What Problem Does This Paper Address?

When a deployed foundation model makes a critical error, exhibits demographic bias, or demonstrates an unexpected emergent skill, who or what is accountable?

Traditional data attribution methods (e.g., influence functions, Data Shapley) attribute behaviors to individual data points. However, they typically evaluate an "average model" under rigid assumptions (convexity, convergence, vanilla SGD), ignoring how multi-stage training evolves over time. Meanwhile, literature on learning dynamics describes parameter trajectories but lacks a principled framework to assign causal credit or blame to specific training phases.

The authors bridge this gap by answering a counterfactual question: How would the model's test performance change if updates from a specific training stage had been omitted?

Methods

The authors frame training as a sequence of causal interventions using the potential outcomes framework:

  • Causal Formulation: Let treatment $T=1$ denote executing training stage $S$, and $T=0$ denote skipping it. The individual treatment effect on model performance $\gamma(x, \theta)$ is $\tau_{K,S} = \gamma(x, \theta_K(1)) - \gamma(x, \theta_K(0))$.
  • First-Order Propagator Derivation: To avoid computationally prohibitive model retraining, the authors define an interpolated trajectory $\epsilon \in [0, 1]$ and take a first-order Taylor expansion around the observed path ($\epsilon = 1$).
  • Incorporating Optimization Dynamics: Unlike prior work limited to vanilla SGD, the state vector tracks both model parameters $\theta$ and momentum velocity $v$. The perturbation propagates forward through a recursive $2p \times 2p$ block matrix $M_k$ that explicitly models learning rate $\eta_k$, momentum $\mu$, and weight decay $\lambda$.
  • Reusable Stage Embeddings: The estimated effect on performance decomposes into: $$\hat{\tau}_{K,t} = \nabla_\theta \gamma(x, \theta_K)^\top E_t$$ Here, $E_t$ is a "stage embedding" capturing parameter differences propagated to the end of training. $E_t$ is computed once during training and can evaluate any downstream test instance or metric via a fast inner product.

  • Approximations for Scale: The authors use the Generalized Gauss-Newton (GGN) matrix for matrix-free Hessian-vector products and restrict heavy computations to the model's final layers.

Datasets

The authors evaluate their method on established vision and language benchmarks (no new datasets were introduced):

  • MNIST: Multi-Layer Perceptrons (MLPs) evaluated against ground-truth model retraining across synthetic interventions (data insertions, label corruption, and rotational domain shifts).
  • CelebA (via WILDS): ResNet-18 predicting hair color (blonde), where hair color is spuriously correlated with gender.
  • CivilComments (via WILDS): Fine-tuned GPT-2 predicting comment toxicity, where toxic labels spuriously correlate with demographic/religious identity terms (e.g., "Christian").

Key Insights and Ideas

  • Decoupled Evaluation: By separating the training dynamics term ($E_t$) from the test metric gradient ($\nabla_\theta \gamma$), practitioners only compute the training trace once. Auditing new test samples or custom objectives requires only a single dot product.
  • Hyperparameter Sensitivity: The attribution score (AA-Score) quantitatively reflects optimization hyperparameters: lower learning rates and momentum decrease attribution magnitudes, while heavy weight decay dampens data-driven learning signals.
  • Auditing Spurious Correlations and Label Noise: By computing attribution against a confounding attribute versus the true target label, the method identifies specific training steps that feed "shortcut learning." Skipping these steps during retraining removes the bias while preserving or boosting core task performance.
  • Data Debugging: In CelebA, the framework ranked a training instance of a blonde male labeled as "not blonde" as having the single most negative effect on blonde prediction, correctly isolating label noise without human supervision.

Key Results

  • Faithfulness to Gold-Standard Retraining: On MNIST, the estimated AA-Score achieved an average Pearson correlation of 0.7314 with actual counterfactual retraining across five settings (reaching 0.9712 for single-instance insertion and 0.7276–0.8430 across rotational distribution shifts).
  • Mitigating Spurious Correlations via Targeted Stage Removal:
  • CelebA: Retraining without the top confounding steps increased real target accuracy from 91.72% to 93.85% while reducing reliance on the spurious gender attribute from 55.01% to 51.87%.
  • CivilComments: Removing toxic/identity shortcut steps improved toxicity classification accuracy from 65.70% to 66.60% and reduced confounding demographic correlation from 47.80% to 46.90%.

Limitations and Future Work

  • First-Order Approximation: The framework relies on linearizing training dynamics, which can lose accuracy across long training runs or when non-linear, higher-order interactions dominate.
  • Computational Footprint: Propagating $2p \times 2p$ states is prohibitive for massive parameters, necessitating layer-wise heuristics (e.g., only analyzing the final classification head).
  • Scale of Foundation Models: Experiments were conducted on relatively small architectures (MLPs, ResNet-18, GPT-2 head tuning); scaling the technique to trillion-token pretraining runs remains an open challenge.
  • Assumed Transparency: The method requires access to complete, granular optimizer states and trajectories, which are rarely shared for commercial, closed-source models.

Summary updated

Abstract

As foundation models (FMs) advance toward artificial general intelligence with increasingly complex training pipelines—involving pretraining, fine-tuning rounds, and subsequent adaptation or alignment—ensuring their reliable and responsible development becomes critical. A key challenge in this context is accountability: when a deployed model exhibits concerning or beneficial behaviors, which training stage is responsible, and to what extent? We pose the problem of \textbf{accountability attribution}, which aims to trace model behavior back to specific stages of the training process, enabling transparency and auditability essential for responsible AI development. To address this, we propose a general framework that answers counterfactual questions about stage effects: \textit{how would the model's behavior have changed if the updates from a training stage had not been executed?}. Within this framework, we introduce estimators based on first-order approximations that efficiently quantify the stage effects without retraining. Our estimators account for both the training data and key aspects of optimization dynamics, including learning rate schedules, momentum, and weight decay. Through experiments on high-stakes domains like medicine and finance, we demonstrate that our approach identifies training stages accountable for specific behaviors, offering a practical tool for ensuring reliable and responsible foundation model development.
BibTeX
@inproceedings{zhang2026accountability,
  author = {Shichang Zhang and Hongzhe Du and Karim Saraipour and Jiaqi W. Ma and Himabindu Lakkaraju},
  title = {{Accountability Attribution: Tracing Model Behavior to Training Processes}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=wfHCePoAed},
}
Figure