Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems

Shichang Zhang, Hongguang Du, Jiaqi W. Ma, Himabindu Lakkaraju

Abstract

Modern AI systems are typically developed through multiple stages—pretraining, fine-tuning rounds, and subsequent adaptation or alignment—where each stage builds on the previous ones and updates the model in distinct ways. This raises a critical question of accountability: when a deployed model succeeds or fails, which stage is responsible, and to what extent? We pose the accountability attribution problem for tracing model behavior back to specific stages of the model development process. To address this challenge, we propose a general framework that answers counterfactual questions about stage effects: how would the model's behavior have changed if the updates from a particular stage had not occurred? Within this framework, we introduce estimators that efficiently quantify stage effects without retraining the model, accounting for both the data and key aspects of model optimization dynamics, including learning rate schedules, momentum, and weight decay. We demonstrate that our approach successfully quantifies the accountability of each stage to the model's behavior. Based on the attribution results, our method can identify and remove spurious correlations learned during image classification and text toxicity detection tasks that were developed across multiple stages. Our approach provides a practical tool for model analysis and represents a significant step toward more accountable AI development.

Introduction

Modern AI development is rarely a single-step process. Instead, it resembles a relay race where a model is passed through various stages: large-scale pretraining, domain-specific fine-tuning, and human-alignment rounds like RLHF. While this modular approach is powerful, it creates a significant accountability gap. When a deployed model exhibits a specific behavior—whether it is a correct answer, a racial bias, or a reliance on spurious correlations—it is difficult to determine which stage of the training process is responsible.

Conceptual overview of multi-stage development and attribution.

The Accountability Attribution Problem

The core challenge lies in tracing a model's final behavior back to specific temporal stages. Existing data attribution tools often tell us which individual data points were influential, but they frequently treat models as static functions of a dataset, ignoring the dynamic nature of training. To address this, we define the accountability attribution problem: quantifying the influence of each developmental stage on the final model's output by accounting for the entire training trajectory.

Methodology: The AA-Score

We propose a general framework based on the potential outcomes formalism from causal inference. We ask the counterfactual question: "How would the model's behavior on this specific test case change if the updates from Stage X had never occurred?"

To make this practical without the prohibitive cost of retraining the model for every possible "what-if" scenario, we introduce the AA-Score (Accountability Attribution Score). This estimator leverages a first-order Taylor expansion to approximate the effect of removing a stage. Crucially, the AA-Score is optimization-aware. Unlike previous methods that assume simple Gradient Descent, our framework explicitly incorporates the effects of learning rate schedules, momentum, and weight decay, which are fundamental to how modern models learn and retain information over time.

Furthermore, the method produces Stage Embeddings. These are computed once for each training stage and can be reused to analyze any number of different test inputs or performance metrics. This makes the tool highly efficient for post-hoc analysis, as the heavy lifting of processing the training trajectory is decoupled from the specific questions a developer might ask about the model's behavior.

Experimental Validation

We validated our approach across a diverse set of tasks and modalities, including MNIST digit classification, CelebA facial attribute detection, and toxicity detection in the CivilComments dataset. We also evaluated geographic generalization using a multi-site chest X-ray task.

Performance effect estimation across various training scenarios and optimizer settings.

As shown in Figure 3, our estimator accurately tracks the impact of specific training events. For instance, it can pinpoint the "Stage of Insertion" for specific data points (a) or identify stages where mislabeling occurred (b). Furthermore, the framework demonstrates how different optimizer settings—such as learning rate (g), momentum (h), and weight decay (i)—alter the attribution of accountability across the training trajectory.

Key Results:

  • High Fidelity: The AA-Score showed a very high correlation (averaging 0.9456 on MNIST and up to 0.98 on other tasks) with the "ground truth" results obtained by actually retraining the models from scratch.
  • Bias Mitigation: In the CivilComments task, we identified stages that increased the model's likelihood of flagging comments based on demographic terms. By skipping the most biased stages identified by our tool, we significantly reduced the model's reliance on these features without destroying overall performance.
  • Distribution Shift Tracking: On the multi-site X-ray task, the tool correctly identified that the stage using data from a specific location (e.g., Spain) provided the most "credit" for the model's performance on test data from that same location.

Discussion and Future Work

While the AA-Score provides a powerful diagnostic tool, it relies on linear approximations that may lose accuracy in highly non-linear "chaotic" training phases. Additionally, computing the necessary mathematical components for massive foundation models remains an engineering challenge. Future work will explore low-rank approximations to scale the framework to models like GPT-4 or Llama-3.

Ultimately, the Accountability Attribution framework provides a practical and efficient tool for model analysis. By moving beyond simple data attribution to process-aware accountability, developers can better understand, debug, and align complex AI systems, representing a significant step toward more transparent and responsible AI development.

Summary updated

Abstract

Modern AI systems are typically developed through multiple stages-pretraining, fine-tuning rounds, and subsequent adaptation or alignment, where each stage builds on the previous ones and updates the model in distinct ways. This raises a critical question of accountability: when a deployed model succeeds or fails, which stage is responsible, and to what extent? We pose the accountability attribution problem for tracing model behavior back to specific stages of the model development process. To address this challenge, we propose a general framework that answers counterfactual questions about stage effects: how would the model's behavior have changed if the updates from a particular stage had not occurred? Within this framework, we introduce estimators that efficiently quantify stage effects without retraining the model, accounting for both the data and key aspects of model optimization dynamics, including learning rate schedules, momentum, and weight decay. We demonstrate that our approach successfully quantifies the accountability of each stage to the model's behavior. Based on the attribution results, our method can identify and remove spurious correlations learned during image classification and text toxicity detection tasks that were developed across multiple stages. Our approach provides a practical tool for model analysis and represents a significant step toward more accountable AI development.
BibTeX
@inproceedings{zhang2026who,
  author = {Shichang Zhang and Hongguang Du and Jiaqi W. Ma and Himabindu Lakkaraju},
  title = {{Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=wfHCePoAed},
}
Figure