Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?
Lists
Only you see this
Abstract
Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse. Moreover, ratio-based clipping can fail in fully on-policy training: the importance ratio stays at 1, so the ratio-based trust region constraint never activates. Under standard training settings where GRPO is stable, LGPO achieves comparable training stability and peak performance while reducing training time by ~18% on average relative to a standard implementation that recomputes old-policy likelihoods. In fully on-policy training, where GRPO fails, LGPO remains stable, enabling more efficient and robust LLM RL post-training across training regimes.
BibTeX
@inproceedings{zhang2026is,
author = {Shuibai Zhang and Junhyuck Kim and Gyeongman Kim and Jaewoong Cho},
title = {{Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?}},
booktitle = {SPOT @ ICLR},
year = {2026},
url = {https://openreview.net/forum?id=eiOwyAa2O1},
}Loading related content...