SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

Dayang Liang, Lang Feng, Bo An, Yunlong LiuarXiv 2026 · arXiv

TL;DR

SAPO (Single-Rollout Autoregressive Policy Optimization) is a lightweight reinforcement learning framework for LLM agents that embeds policy, state-value, and action-value functions within a single autoregressive model. By eliminating both the massive memory footprint of a separate critic network and the heavy compute cost of multi-rollout group sampling, SAPO cuts training runtime by 33.2% compared to PPO while significantly boosting task success rates.

What Problem Does This Paper Address?

Post-training large language model (LLM) agents using Reinforcement Learning (RL) typically relies on two paradigms, both of which suffer from major efficiency bottlenecks. Traditional Actor-Critic methods like Proximal Policy Optimization (PPO) use a separate critic model to estimate values and assign credit, but maintaining a policy-scale critic doubles the memory and compute overhead.

To circumvent this, critic-free "group-relative" methods like GRPO sample multiple trajectories per prompt to calculate relative advantages. However, GRPO introduces new challenges: weak token-level credit assignment (since rewards are applied uniformly across long trajectories), "advantage collapse" when all sampled rollouts fail under sparse rewards, and a costly compute trade-off requiring large sample sizes to reduce gradient noise. SAPO aims to achieve the best of both worlds: explicit value-based credit assignment without a separate critic or multiple rollouts.

Methods

SAPO introduces a single-stream architecture where the policy, state value ($V$), and action value ($Q$) share the exact same transformer backbone.

  • Causal Boundary Readouts: The framework leverages the natural causal order of language generation. Right before generating an action, the model reads out the state value $V(s_t)$. Right after generating the action, it reads out the action value $Q(s_t, a_t)$.
  • Two-Token Value Basis: Instead of adding separate heads or adapters, SAPO reserves two existing vocabulary tokens ($w^+$ and $w^-$) to act as a scalar readout mechanism based on their logit differences.
  • Single-Rollout GAE & SARSA: It uses a trajectory-level Generalized Advantage Estimator (GAE) with $\lambda$-returns to assign turn-level credit from just a single rollout. The state value is optimized via $\lambda$-returns, while the action value is trained using an auxiliary on-policy SARSA objective.
  • Batch Normalization: Instead of normalizing advantages within a single prompt's group (like GRPO), SAPO normalizes turn-level advantages across the entire training batch.
SAPO Framework

Datasets

The authors evaluated SAPO using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct on two challenging multi-turn interactive benchmarks:

  • ALFWorld: An embodied household environment requiring text-based reasoning and decision-making across six task categories (Pick, Look, Clean, Heat, Cool, and Pick2).
  • WebShop: An interactive web-shopping environment containing nearly 1.1 million products where agents must follow user instructions to navigate, search, and purchase items.

Key Insights and Ideas

  • Alignment with Causal Flow: The core "aha" moment is that actor-critic information flow perfectly mirrors language generation. A state value should summarize everything before an action, and an action value should condition on the action after it's generated. Causal masking naturally enforces this separation without needing separate networks.
  • Vocabulary as a Value Head: Reusing existing vocabulary tokens as a value basis avoids adding role-specific adapters or heads, preventing policy-value interference and keeping the architecture incredibly clean.
  • Batch vs. Group Normalization: Shifting normalization from a single prompt's group to the entire training batch allows the model to learn effectively from a single rollout per prompt, completely bypassing the advantage collapse problem seen in GRPO when all rollouts for a hard prompt fail.

Key Results

  • Performance Boost: SAPO outperforms standard PPO and GRPO by an average of +15.1 and +12.1 percentage points across tasks. For instance, with Qwen2.5-7B, SAPO achieved a 94.0% success rate on ALFWorld and an 88.6 score on WebShop, outperforming both proprietary models like GPT-4o and specialized RL baselines.
  • Efficiency Gains: It completely eliminates the memory footprint of a separate critic model.
  • Runtime Reduction: SAPO reduces per-iteration runtime by 33.2% compared to PPO, primarily driven by faster trajectory generation and the structural removal of the independent critic update phase.
Figure 2: Per-iteration runtime breakdown of PPO and SAPO on ALFWorld using Qwen2.5-1.5B. The vertical axis is shown on a logarithmic scale to accommodate the large variation in module runtimes. N/A indicates that SAPO does not require the corresponding value-model or critic-update module. Overall, SAPO reduces the measured per-iteration runtime from 451.2 s to 301.4 s, corresponding to a 33.2% reduction compared with PPO.

Limitations and Future Work

While SAPO achieves state-of-the-art aggregate performance, the authors note it is not uniformly the top performer across every single sub-category (e.g., certain baselines like GiGPO slightly outperform it on specific sub-tasks like Look or Pick2 in the 1.5B variant). Additionally, the paper does not deeply explore how this single-backbone approach scales to ultra-large models (e.g., 70B+) or how the shared representations handle highly divergent multi-task reward structures. Future work could investigate the limits of parameter sharing when scaling up model sizes or task complexity.

Summary updated

Abstract

Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks. Despite their success, recent studies revealed three limitations: (1) Lack explicit value generalization and effective temporal credit assignment; (2) Suffer from potential advantage collapse in long-horizon complex tasks; (3) Require a costly trade-off between sampling budget and policy performance. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework in which the policy and value functions share a single autoregressive backbone. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on-policy SARSA objectives. To robustly estimate the contribution of each turn, we further introduce a trajectory-level generalized advantage estimator that combines lambda-returns with batch normalization. Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per-iteration runtime by 33.2% over PPO. Project page: https://github.com/dy-liang/SAPO.
BibTeX
@misc{liang2026sapo,
  author = {Dayang Liang and Lang Feng and Bo An and Yunlong Liu},
  title = {{SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning}},
  year = {2026},
  eprint = {2608.19842},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.19842},
}
Figure