Exploration of Reinforcement Learning in Generative Recommendation Models

Pengfei Zheng, Wuchao Li, Shiyao Wang, Honghui Bao, Hongyi Xie, Jiaxin Deng, Xingmei Wang, Jinghao Zhang, Yufei Ye, Ruiming Tang, Guorui Zhou, Han Li, Kun Gai, Defu LianarXiv 2026 · arXiv

TL;DR

Generative recommendation models offer a unified alternative to messy multi-stage pipelines, but training them with Reinforcement Learning (RL) is notoriously unstable. This paper introduces Gradient-Bounded Policy Optimization (GBPO), a new RL objective that prevents training collapse by bounding gradients and ensuring the model stops "over-learning" once it becomes confident, leading to state-of-the-art stability and accuracy across major benchmarks.

What Problem Does This Paper Address?

Traditional industrial recommendation systems are like a relay race: one model retrieves a few thousand items, another ranks them, and a third re-ranks them for diversity. Because each stage is trained separately, their objectives often clash, making end-to-end optimization nearly impossible.

Generative Recommendation solves this by treating recommendation as a translation task: given a user's history, the model autoregressively "generates" the ID of the next item. While this allows us to use RL to optimize the whole process at once, standard RL algorithms (like PPO or GRPO) often fail in this setting. They suffer from "training collapse"—where performance suddenly drops to zero—or they get stuck because their mathematical objectives create "dead zones" where the model stops learning from available data.

Methods

The authors first diagnose the failures of existing methods using a Unified Logit-Gradient View. They found that popular methods like Group Relative Policy Optimization (GRPO) have three structural flaws:

  1. Dead-gradient regions: Clipping mechanisms (intended to keep updates small) often result in zero gradients, wasting valuable training samples.
  2. Unbounded negative updates: When the model is told an item is "bad," the mathematical penalty can explode if the model’s current probability for that item is high, leading to numerical instability.
  3. Confidence insensitivity: Standard ratios don't care how sure the model is. A model might be 99% sure about a correct item but still receive a massive update, which can destabilize the internal representations.

To fix this, they propose Gradient-Bounded Policy Optimization (GBPO). Instead of hard clipping, GBPO uses a stop-gradient normalization mechanism. This ensures:

  • Full Sample Utilization: Every sampled token provides a learning signal (no dead zones).
  • Sign-Symmetric Bounding: Gradients for both "good" and "bad" items are strictly capped, preventing the "negative update explosion."
  • Confidence Weighting: Updates naturally vanish as the model approaches a probability of 1 for positive items or 0 for negative ones, mimicking the stable behavior of Binary Cross-Entropy (BCE).

Datasets

The researchers evaluated GBPO on four real-world recommendation benchmarks:

  • Amazon Beauty, Toys, and Sports: Standard datasets for sequential recommendation.
  • Steam: A dataset of game reviews and interactions used to test generalization outside the Amazon ecosystem.

They tested GBPO across two different generative backbones: TIGER (which uses semantic IDs) and HSTU (which predicts item IDs directly).

Key Insights and Ideas

  • The "BCE Connection": The authors realized that the most stable way to update a policy is to make its gradient look like a weighted version of the standard Binary Cross-Entropy loss. By forcing the RL objective to behave like BCE, they inherited its natural stability.
  • Clipping is a Double-Edged Sword: While clipping is meant to prevent "policy shock," in the constrained space of item IDs, it frequently creates "dead gradients" that starve the model of information.
  • Negative Dominance: The paper highlights that training instability is often driven by a small subset of "negative" samples (items the model shouldn't recommend) that generate massive, unbounded gradients. GBPO’s symmetric bounding is the "secret sauce" that prevents this.

Key Results

  • Superior Stability: In head-to-head tests on the Amazon Sports dataset, methods like GRPO and GSPO collapsed (performance dropped to near zero) within 80 epochs. GBPO maintained a high, stable performance plateau throughout the entire training run.
  • Accuracy Gains: GBPO consistently outperformed supervised learning baselines and other RL methods. For example, on the Toys dataset, TIGER trained with GBPO improved HitRate@10 by over 2 percentage points compared to the supervised version.
  • Backbone Agnostic: The method worked equally well whether the model was generating complex semantic tokens (TIGER) or simple item IDs (HSTU), suggesting it addresses a fundamental flaw in RL optimization rather than a specific architecture quirk.

Limitations and Future Work

  • Computational Overhead: Like all RL methods, GBPO requires a "generate-evaluate-update" loop. Generating multiple candidates for every user history is significantly more expensive than standard supervised training.
  • Reward Quality: The model relies on a "teacher" model (like a Deep Interest Network) to provide rewards. If the teacher is biased or inaccurate, GBPO will simply become very good at mimicking those flaws.
  • Offline vs. Online: The study is strictly offline. The authors note that moving to online learning (with real user feedback) will introduce new challenges like delayed rewards and non-stationary data that GBPO hasn't yet been tested against.

Summary updated

Abstract

Traditional cascaded recommender systems balance accuracy and efficiency over large item spaces, but misaligned stage-wise objectives hinder end-to-end optimization. Generative recommendation unifies these stages in an autoregressive framework, providing a natural interface for reinforcement learning (RL) to optimize the entire generation process. However, directly applying RL often leads to unstable optimization and degraded recommendation quality. We revisit Group Relative Policy Optimization (GRPO) and other ratio-based objectives through a unified logit-gradient view. We identify three issues: (i) clipped surrogates create dead-gradient regions, under-utilizing samples; (ii) negative-advantage updates can become effectively unbounded as importance ratios grow, allowing a small subset of negatives to dominate optimization; and (iii) ratio-based control lacks sensitivity to absolute policy confidence, so updates on high-confidence cases may fail to vanish. Motivated by this diagnosis, we propose \emph{Gradient-Bounded Policy Optimization} (GBPO), a simple objective with stop-gradient normalization that is simultaneously (a) fully gradient-active, (b) sign-symmetrically bounded for both positive and negative updates, and (c) explicitly confidence-weighted under current policy probabilities. This ensures that updates gracefully vanish for $π\!\to\!1$ positives and $π\!\to\!0$ negatives. Across the evaluated settings, GBPO achieves competitive recommendation performance with more stable training than competing RL methods.
BibTeX
@misc{zheng2026exploration,
  author = {Pengfei Zheng and Wuchao Li and Shiyao Wang and Honghui Bao and Hongyi Xie and Jiaxin Deng and Xingmei Wang and Jinghao Zhang and Yufei Ye and Ruiming Tang and Guorui Zhou and Han Li and Kun Gai and Defu Lian},
  title = {{Exploration of Reinforcement Learning in Generative Recommendation Models}},
  year = {2026},
  eprint = {2611.19742},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19742},
}
Figure