Who Gets the Reward, Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents

Chih-Hsuan Yang, Tanwi Mallick, Le Chen, Krishnan Raghavan, Azton Wells, Amal Gueroudji, Ian T. Foster, Rajeev Thakur

TL;DR

This paper introduces a novel framework for training multi-agent systems powered by Large Language Models (LLMs). It bridges the gap between system-level evaluation and individual agent learning by generating "signed, credit-conserving" training signals, ensuring that both successful and failed episodes contribute to more robust and cooperative agent behavior.

What Problem Does This Paper Address?

Training multi-agent systems (MAS) composed of LLMs presents a significant challenge. While LLMs excel at complex tasks, current training methods struggle to translate overall system performance (like success or failure) into actionable feedback for individual agents and their specific messages. This makes it difficult to ensure agents learn to cooperate effectively, avoid redundant work, and recover from errors. Existing approaches often focus on single agents or rely on assumptions that don't hold for high-dimensional, text-based LLM interactions, leading to potential issues like credit hoarding, misdirected learning, and fragile collaboration.

Methods

The paper proposes a two-pronged approach to generate training signals:

  • Success Route (System → Agent → Message):
    • Shapley Values for Agent Credit: For successful runs, Shapley values are used to fairly attribute the system's reward (Rsys) to individual agents. This method, rooted in cooperative game theory, measures an agent's marginal contribution across all possible coalitions, ensuring credit is given for unique contributions and penalizing sabotage.
  • Process Reward Modeling (PRM) for Message Refinement: The agent-level credit is then further decomposed into message-level rewards. This is achieved by using a "judge" (potentially another LLM) to assign signed labels (+1 for helpful, -1 for harmful, 0 for neutral) to each message. These labels, combined with the agent's Shapley credit, create fine-grained, signed rewards for each message, encouraging informative actions and penalizing redundancy.
  • Failure Route (First-Error Localization → Preferences):
    • First-Error Localization: In failed episodes (Rsys = 0), instead of trying to divide a zero reward, the system identifies the earliest message that caused the failure.
  • Repair-Aware Preferences: This identified error message is then used to construct contrastive training pairs. These pairs highlight the harmful message versus a preferred alternative (e.g., a corrected version), allowing for preference-based learning (like DPO/GRPO) that not only penalizes the error but also rewards subsequent repair attempts.

Datasets

Not discussed in the paper. The framework is presented as a theoretical foundation and is intended for empirical validation in future work.

Key Insights and Ideas

  • Bridging System and Agent Levels: The core innovation is a principled way to connect high-level system evaluation to low-level agent and message supervision, a critical gap for LLM MAS.
  • Signed, Credit-Conserving Signals: The generated signals are "signed" (can be positive or negative, indicating helpfulness or harm) and "credit-conserving" (sum up to the total system reward), ensuring a fair and bounded learning objective.
  • Shapley Values for Fair Attribution: Using Shapley values for agent credit assignment is a robust way to ensure agents are rewarded for their unique contributions, not just for being present or for actions that are easily replicated.
  • Failure as a Learning Opportunity: The failure route is particularly novel. By localizing the first error and then rewarding repair attempts, the framework turns failures into valuable learning signals, unlike methods that simply mark all subsequent steps as invalid.
  • Unified Framework: The paper unifies both success and failure scenarios into a single, coherent pipeline, making the training process more comprehensive.

Key Results

The paper presents a theoretical framework and does not report empirical results. The key "results" are the theoretical guarantees of the framework:

  • Efficiency/Credit Conservation: The sum of agent rewards equals the system reward, and the sum of message rewards for an agent equals its agent reward.
  • Boundedness: Message rewards are bounded, and optional clipping can enforce a fixed range for stability.
  • Anti-Competition: The framework discourages agents from sabotaging each other or duplicating work, as this does not increase their individual credit.
  • Repair-Awareness: The failure route explicitly distinguishes between errors and subsequent repair attempts, rewarding the latter.

Limitations and Future Work

  • Computational Cost of Shapley Values: Exact Shapley value computation is exponential in the number of agents. The paper suggests Monte Carlo permutation sampling as a practical approximation and mentions that surrogate models can be trained to further reduce costs.
  • Reliability of Judges: The effectiveness of the PRM-style judges (LLMs or other classifiers) for assigning message labels is a potential concern. The authors note that their judges need only assess local, role-specific alignment, reducing the burden and expertise required.
  • Heterogeneous Teams: The current framework assumes homogeneous teams or role-specialized agents. Extending it to fully heterogeneous teams (different foundation models for different roles) requires careful design of baseline policies for fair Shapley attribution.
  • Empirical Validation: The paper is primarily theoretical. The authors state that their next step is to conduct experiments to empirically validate the framework, quantify trade-offs, and compare it against existing baselines.

Summary updated

Abstract

Large Language Models (LLMs) in multi-agent systems (MAS) have shown promise for complex tasks, yet current training methods lack principled ways to connect system-level evaluation with agent-level and message-level learning. We propose a theoretical framework that unifies cooperative game-theoretic attribution with process reward modeling to transform system evaluation into agent credit and then into response-level signals. Unlike prior approaches that rely only on attribution (e.g., Shapley) or step-level labels (e.g., PRM), our method produces local, signed, and credit-conserving signals. In success cases, Shapley-based credit assignment fairly allocates outcomes across agents and is refined into per-message rewards that promote cooperation while discouraging redundancy or sabotage. In failure cases, first-error localization yields repair-aware preferences that penalize harmful steps while rewarding corrective attempts. The resulting signals are bounded, cooperative, and directly compatible with reinforcement-based or preference-based post-training, providing a unified and auditable pathway from global evaluation to local supervision in LLM multi-agent training. Our contribution is conceptual: we present a theoretical foundation and training signals, leaving empirical validation for future work.
BibTeX
@misc{yang2025who,
  author = {Chih-Hsuan Yang and Tanwi Mallick and Le Chen and Krishnan Raghavan and Azton Wells and Amal Gueroudji and Ian T. Foster and Rajeev Thakur},
  title = {{Who Gets the Reward, Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents}},
  year = {2025},
  eprint = {2511.10687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.10687},
}
Figure