TL;DR
Outcome-Mass Alignment (OMA) fixes a subtle but pervasive accounting error in multi-turn LLM agent training where longer trajectories (often failures) accidentally dominate the learning process. By explicitly aligning outcome statistics and objective mass at the trajectory level, OMA improves success rates by up to 9.4 percentage points in complex environments like ALFWorld.
What Problem Does This Paper Address?
When training LLM agents for multi-turn tasks—like navigating a house in ALFWorld or shopping in WebShop—researchers often use "stepwise" optimization. To keep long interaction histories manageable, each turn of a trajectory is treated as a separate row in a training batch.
However, this creates a hidden "outer-accounting" problem. If a failed trajectory takes 50 turns to hit a limit while a successful one takes only 10, the failed trajectory appears five times more often in the training data. This leads to two issues:
- Outcome-statistics multiplicity: The average "success rate" used to calculate relative advantages is skewed because long trajectories are over-counted.
- Objective-mass imbalance: The gradient updates are dominated by longer trajectories simply because they contain more tokens, regardless of whether those turns were actually informative.
As shown in the figure below, a failed trajectory can account for nearly 80% of the statistical and optimization weight in a simple pair comparison, purely due to its length.
Methods
The authors introduce Outcome-Mass Alignment (OMA), a two-component framework that restores balance without changing the underlying reinforcement learning algorithm (like GRPO).
- Outcome Alignment (OA): Instead of calculating group statistics (like mean and variance of rewards) over every row in a batch, OA calculates them over unique source trajectories. This ensures that a single 50-turn failure doesn't pull the baseline down more than a single 10-turn success.
- Mass Alignment (MA): This component ensures every trajectory has an equal "vote" in the final gradient. It aggregates the token-level objectives within each trajectory first, then averages them across the group. This assigns each source trajectory a total outer mass of $1/K$, regardless of its length.
Datasets
The researchers evaluated OMA using Qwen2.5-1.5B/3B-Instruct models on:
- ALFWorld: A benchmark for embodied agents performing household tasks. Here, failures are typically much longer than successes.
- WebShop: A goal-directed web navigation task.
- WebShop-SOL: A "success-oriented" variant of WebShop introduced by the authors to test OMA in settings where the agent is encouraged to keep trying until it succeeds.
Key Insights and Ideas
- Implicit Weighting is Everywhere: Any stepwise training pipeline that doesn't explicitly account for trajectory length is making an implicit choice about which trajectories matter most.
- Coupling is the Catalyst: OMA is most effective when trajectory length is strongly "coupled" with the outcome. In ALFWorld, where long trajectories are almost always failures, OMA prevents the model from being overwhelmed by "learning what not to do" at the expense of successful patterns.
- Orthogonality: OMA doesn't replace credit assignment (deciding which step caused a win). Instead, it manages trajectory-level influence, making it compatible with various optimizers like GRPO, GiGPO, and GSPO.
Key Results
- ALFWorld Performance: OMA improved endpoint success rates by 7.0–9.4 percentage points compared to standard GRPO.
- WebShop-SOL: In the success-oriented setting, OMA boosted the normalized Area Under the Curve (nAUC) for success from 0.399 to 0.511.
- Gradient Geometry: Using "frozen-batch" audits, the authors proved that Mass Alignment significantly alters the gradient direction and norm when length asymmetry is high, preventing the "mass" of long trajectories from drowning out shorter ones.
Limitations and Future Work
- Condition Dependency: The benefits of OMA are less pronounced (or even reversed) in environments where trajectory length and outcome are not systematically linked.
- Dynamic Coupling: The relationship between length and success can change as the model learns. A strategy that works early in training might need adjustment as the agent becomes more efficient.
- Scale: While tested on 1.5B and 3B models, the impact on much larger frontier models remains an open question for future empirical study.
Summary updated