TL;DR
SAPO (Single-Rollout Autoregressive Policy Optimization) is a lightweight reinforcement learning framework for LLM agents that embeds policy, state-value, and action-value functions within a single autoregressive model. By eliminating both the massive memory footprint of a separate critic network and the heavy compute cost of multi-rollout group sampling, SAPO cuts training runtime by 33.2% compared to PPO while significantly boosting task success rates.
What Problem Does This Paper Address?
Post-training large language model (LLM) agents using Reinforcement Learning (RL) typically relies on two paradigms, both of which suffer from major efficiency bottlenecks. Traditional Actor-Critic methods like Proximal Policy Optimization (PPO) use a separate critic model to estimate values and assign credit, but maintaining a policy-scale critic doubles the memory and compute overhead.
To circumvent this, critic-free "group-relative" methods like GRPO sample multiple trajectories per prompt to calculate relative advantages. However, GRPO introduces new challenges: weak token-level credit assignment (since rewards are applied uniformly across long trajectories), "advantage collapse" when all sampled rollouts fail under sparse rewards, and a costly compute trade-off requiring large sample sizes to reduce gradient noise. SAPO aims to achieve the best of both worlds: explicit value-based credit assignment without a separate critic or multiple rollouts.
Methods
SAPO introduces a single-stream architecture where the policy, state value ($V$), and action value ($Q$) share the exact same transformer backbone.
- Causal Boundary Readouts: The framework leverages the natural causal order of language generation. Right before generating an action, the model reads out the state value $V(s_t)$. Right after generating the action, it reads out the action value $Q(s_t, a_t)$.
- Two-Token Value Basis: Instead of adding separate heads or adapters, SAPO reserves two existing vocabulary tokens ($w^+$ and $w^-$) to act as a scalar readout mechanism based on their logit differences.
- Single-Rollout GAE & SARSA: It uses a trajectory-level Generalized Advantage Estimator (GAE) with $\lambda$-returns to assign turn-level credit from just a single rollout. The state value is optimized via $\lambda$-returns, while the action value is trained using an auxiliary on-policy SARSA objective.
- Batch Normalization: Instead of normalizing advantages within a single prompt's group (like GRPO), SAPO normalizes turn-level advantages across the entire training batch.
Datasets
The authors evaluated SAPO using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct on two challenging multi-turn interactive benchmarks:
- ALFWorld: An embodied household environment requiring text-based reasoning and decision-making across six task categories (Pick, Look, Clean, Heat, Cool, and Pick2).
- WebShop: An interactive web-shopping environment containing nearly 1.1 million products where agents must follow user instructions to navigate, search, and purchase items.
Key Insights and Ideas
- Alignment with Causal Flow: The core "aha" moment is that actor-critic information flow perfectly mirrors language generation. A state value should summarize everything before an action, and an action value should condition on the action after it's generated. Causal masking naturally enforces this separation without needing separate networks.
- Vocabulary as a Value Head: Reusing existing vocabulary tokens as a value basis avoids adding role-specific adapters or heads, preventing policy-value interference and keeping the architecture incredibly clean.
- Batch vs. Group Normalization: Shifting normalization from a single prompt's group to the entire training batch allows the model to learn effectively from a single rollout per prompt, completely bypassing the advantage collapse problem seen in GRPO when all rollouts for a hard prompt fail.
Key Results
- Performance Boost: SAPO outperforms standard PPO and GRPO by an average of +15.1 and +12.1 percentage points across tasks. For instance, with Qwen2.5-7B, SAPO achieved a 94.0% success rate on ALFWorld and an 88.6 score on WebShop, outperforming both proprietary models like GPT-4o and specialized RL baselines.
- Efficiency Gains: It completely eliminates the memory footprint of a separate critic model.
- Runtime Reduction: SAPO reduces per-iteration runtime by 33.2% compared to PPO, primarily driven by faster trajectory generation and the structural removal of the independent critic update phase.
Limitations and Future Work
While SAPO achieves state-of-the-art aggregate performance, the authors note it is not uniformly the top performer across every single sub-category (e.g., certain baselines like GiGPO slightly outperform it on specific sub-tasks like Look or Pick2 in the 1.5B variant). Additionally, the paper does not deeply explore how this single-backbone approach scales to ultra-large models (e.g., 70B+) or how the shared representations handle highly divergent multi-task reward structures. Future work could investigate the limits of parameter sharing when scaling up model sizes or task complexity.
Summary updated