Outcome-Mass Alignment: Controlling Length-Dependent Trajectory Influence in Multi-Turn LLM Agent Reinforcement Learning

Dawei Zhang, Yubin Huangfu, Jessica Fan, Boyu Wei, Likang Wu, Hongke Zhao, Ming He, Zhu TengarXiv 2026 · arXiv

TL;DR

Outcome-Mass Alignment (OMA) fixes a subtle but pervasive accounting error in multi-turn LLM agent training where longer trajectories (often failures) accidentally dominate the learning process. By explicitly aligning outcome statistics and objective mass at the trajectory level, OMA improves success rates by up to 9.4 percentage points in complex environments like ALFWorld.

What Problem Does This Paper Address?

When training LLM agents for multi-turn tasks—like navigating a house in ALFWorld or shopping in WebShop—researchers often use "stepwise" optimization. To keep long interaction histories manageable, each turn of a trajectory is treated as a separate row in a training batch.

However, this creates a hidden "outer-accounting" problem. If a failed trajectory takes 50 turns to hit a limit while a successful one takes only 10, the failed trajectory appears five times more often in the training data. This leads to two issues:

  1. Outcome-statistics multiplicity: The average "success rate" used to calculate relative advantages is skewed because long trajectories are over-counted.
  2. Objective-mass imbalance: The gradient updates are dominated by longer trajectories simply because they contain more tokens, regardless of whether those turns were actually informative.

As shown in the figure below, a failed trajectory can account for nearly 80% of the statistical and optimization weight in a simple pair comparison, purely due to its length.

Figure 1: Expansion-induced trajectory reweighting under stepwise materialization. (a) Same-task, same-checkpoint ALFWorld rollouts: a 14-turn success and a 50-turn failure, each with one episode outcome. (b) One-response- row-per-turn materialization expands τi into ni rows containing Di valid response tokens. (c) Row-based statistics weight trajectory outcomes by ni, whereas token-mean reduction assigns objective mass by Di.

Methods

The authors introduce Outcome-Mass Alignment (OMA), a two-component framework that restores balance without changing the underlying reinforcement learning algorithm (like GRPO).

  • Outcome Alignment (OA): Instead of calculating group statistics (like mean and variance of rewards) over every row in a batch, OA calculates them over unique source trajectories. This ensures that a single 50-turn failure doesn't pull the baseline down more than a single 10-turn success.
  • Mass Alignment (MA): This component ensures every trajectory has an equal "vote" in the final gradient. It aggregates the token-level objectives within each trajectory first, then averages them across the group. This assigns each source trajectory a total outer mass of $1/K$, regardless of its length.
The OMA framework: OA broadcasts unique trajectory stats to rows, while MA equalizes the loss contribution across trajectories.

Datasets

The researchers evaluated OMA using Qwen2.5-1.5B/3B-Instruct models on:

  • ALFWorld: A benchmark for embodied agents performing household tasks. Here, failures are typically much longer than successes.
  • WebShop: A goal-directed web navigation task.
  • WebShop-SOL: A "success-oriented" variant of WebShop introduced by the authors to test OMA in settings where the agent is encouraged to keep trying until it succeeds.

Key Insights and Ideas

  • Implicit Weighting is Everywhere: Any stepwise training pipeline that doesn't explicitly account for trajectory length is making an implicit choice about which trajectories matter most.
  • Coupling is the Catalyst: OMA is most effective when trajectory length is strongly "coupled" with the outcome. In ALFWorld, where long trajectories are almost always failures, OMA prevents the model from being overwhelmed by "learning what not to do" at the expense of successful patterns.
  • Orthogonality: OMA doesn't replace credit assignment (deciding which step caused a win). Instead, it manages trajectory-level influence, making it compatible with various optimizers like GRPO, GiGPO, and GSPO.

Key Results

  • ALFWorld Performance: OMA improved endpoint success rates by 7.0–9.4 percentage points compared to standard GRPO.
  • WebShop-SOL: In the success-oriented setting, OMA boosted the normalized Area Under the Curve (nAUC) for success from 0.399 to 0.511.
  • Gradient Geometry: Using "frozen-batch" audits, the authors proved that Mass Alignment significantly alters the gradient direction and norm when length asymmetry is high, preventing the "mass" of long trajectories from drowning out shorter ones.
Success curves showing OMA (green) consistently outperforming the Base (black) and individual components (blue/orange) in ALFWorld.

Limitations and Future Work

  • Condition Dependency: The benefits of OMA are less pronounced (or even reversed) in environments where trajectory length and outcome are not systematically linked.
  • Dynamic Coupling: The relationship between length and success can change as the model learns. A strategy that works early in training might need adjustment as the agent becomes more efficient.
  • Scale: While tested on 1.5B and 3B models, the impact on much larger frontier models remains an open question for future empirical study.

Summary updated

Abstract

To keep long interaction contexts tractable, stepwise group-relative training for multi-turn LLM agents materializes each source trajectory into multiple response rows and valid tokens. This expansion creates an outer-accounting problem distinct from within-trajectory credit assignment: row-based outcome statistics can count one episode outcome multiple times, while token-level reduction can assign unequal aggregate objective mass across trajectories. We term these effects outcome-statistics multiplicity and objective-mass imbalance. We introduce Outcome-Mass Alignment (OMA), a two-component method that makes both accounting choices explicit. Outcome Alignment computes outcome statistics over unique source trajectories, while Mass Alignment controls aggregate objective mass after within-trajectory aggregation of optimizer-native token objectives. Neither redesigns the base optimizer's credit assignment, importance ratios, or clipping. Across matched ALFWorld GRPO comparisons at two model scales, where failed trajectories typically expand much more than successful ones, OMA improves matched-horizon endpoint success by 7.0-9.4 percentage points. Ordinary WebShop exhibits weak or reversed expansion-outcome coupling and correspondingly mixed effects, whereas in a success-oriented WebShop setting, WebShop-SOL, OMA improves success nAUC from 0.399 to 0.511. Component ablations and GiGPO and GSPO comparisons further support component separability and cross-optimizer compatibility. These results show that trajectory expansion always defines an implicit weighting rule, but its practical effect depends on how expansion couples to outcomes and optimizer-native updates.
BibTeX
@misc{zhang2026outcome,
  author = {Dawei Zhang and Yubin Huangfu and Jessica Fan and Boyu Wei and Likang Wu and Hongke Zhao and Ming He and Zhu Teng},
  title = {{Outcome-Mass Alignment: Controlling Length-Dependent Trajectory Influence in Multi-Turn LLM Agent Reinforcement Learning}},
  year = {2026},
  eprint = {2611.16816},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16816},
}
Figure