UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution

Gengrui Zhang, Yao Wang, Xiaoshuang Chen, Hongyi Qian, Kaiqiao Zhan, Ben Wang

Abstract

In recent years, reinforcement learning (RL) has emerged as a powerful tool for optimizing long-term rewards in recommender systems. However, industrial systems are typically multi-stage pipelines (e.g., matching, ranking, re-ranking), presenting a challenge for single-agent RL due to varying observation spaces across stages. We propose UNEX-RL, a novel UNidirectional-EXecution-based multi-agent RL framework. We identify two primary challenges in this setting: observation dependency and the cascading effect. To address these, we introduce the Cascading Information Chain (CIC) method to effectively train the system by separating independent and action-dependent observations. Evaluated on public datasets and a production system with over 100 million users, UNEX-RL demonstrated a 0.558% increase in user usage time compared to state-of-the-art single-agent RL methods.

Introduction

Modern industrial recommender systems are not monolithic; they are complex, multi-stage pipelines designed to filter millions of items down to a handful of recommendations. While traditional approaches focus on immediate feedback like clicks, there is a growing shift toward optimizing long-term user engagement metrics, such as session length and retention. While Reinforcement Learning (RL) is naturally suited for such long-term optimization, its application in multi-stage environments is non-trivial. Standard Multi-Agent RL (MARL) often assumes agents act simultaneously, but recommendation pipelines are inherently sequential and unidirectional.

The Challenge of Multi-Stage Systems

In a typical recommendation pipeline, the output of one stage (e.g., the "Matching" stage) becomes the input for the next (e.g., the "Ranking" stage). This funnel-like structure creates a unique environment for RL agents.

The multi-stage recommendation process as a funnel, where each stage refines the candidate set for the next.

This unidirectional execution introduces two major hurdles:

  1. Observation Dependency (OD): Downstream observations are not static; they depend entirely on the actions taken by upstream agents. This makes standard offline training from fixed replay buffers inconsistent.
  2. Cascading Effect (CE): An upstream agent's action affects the policy and performance of all subsequent agents. Ignoring this interaction leads to sub-optimal global performance, as agents may work at cross-purposes.

The UNEX-RL Framework

To overcome these challenges, we propose the UNEX-RL framework. UNEX-RL treats each stage of the recommendation pipeline as an independent agent within a cooperative MARL setup. The core of the framework is the Cascading Information Chain (CIC).

The UNEX-RL architecture illustrating the unidirectional flow of information and the interaction between actors and prediction systems across stages.

Cascading Information Chain (CIC)

Instead of sampling independent transitions from a buffer, CIC "replays" the entire execution sequence during training. Starting from the initial user state, it iteratively calculates the observations and actions for every stage in the pipeline. This ensures that the critic—the component evaluating the actions—receives a consistent and accurate flow of information that accounts for the cascading dependencies between stages.

Variance Reduction Techniques

Industrial recommendation data is notoriously noisy. To stabilize training, we employ two key techniques:

  • Category Quantile Rescale (CQR): This method normalizes rewards (like watch time) by calculating their quantile relative to similar users and items. This strips away inherent user bias (e.g., some users naturally watch more videos) to reveal the true reinforcement signal.
  • Stopping Gradient (SG): To prevent complex cascading gradients from becoming unstable during backpropagation, we focus the learning on the direct impact of an agent's action on the global reward.

Experimental Results

We evaluated UNEX-RL using the KuaiRand public dataset and live A/B testing on the Kwai App, a short-video platform with over 100 million daily active users.

  • Offline Performance: On KuaiRand, UNEX-RL-CIC consistently outperformed single-agent baselines (DDPG, TD3) and standard MARL approaches across metrics like total WatchTime and Session Length.
  • Online Impact: In production A/B tests, UNEX-RL-CIC achieved a 0.953% increase in daily watch time over the baseline and a 0.558% increase over the previous state-of-the-art single-agent RL approach. In large-scale industrial systems, gains of this magnitude are highly significant.

Conclusion

UNEX-RL demonstrates that the sequential, unidirectional nature of recommendation pipelines is a critical factor that must be modeled to achieve optimal long-term rewards. By using the Cascading Information Chain to manage observation dependencies and cascading effects, UNEX-RL enables effective multi-agent cooperation. Future work will look into more complex stage interactions and broader reward signals, such as multi-day user retention and ecosystem health.

Summary updated

Abstract

In recent years, there has been a growing interest in utilizing reinforcement learning (RL) to optimize long-term rewards in recommender systems. Since industrial recommender systems are typically designed as multi-stage systems, RL methods with a single agent face challenges when optimizing multiple stages simultaneously. The reason is that different stages have different observation spaces, and thus cannot be modeled by a single agent. To address this issue, we propose a novel UNidirectional-EXecution-based multi-agent Reinforcement Learning (UNEX-RL) framework to reinforce the long-term rewards in multi-stage recommender systems. We show that the unidirectional execution is a key feature of multi-stage recommender systems, bringing new challenges to the applications of multi-agent reinforcement learning (MARL), namely the observation dependency and the cascading effect. To tackle these challenges, we provide a cascading information chain (CIC) method to separate the independent observations from action-dependent observations and use CIC to train UNEX-RL effectively. We also discuss practical variance reduction techniques for UNEX-RL. Finally, we show the effectiveness of UNEX-RL on both public datasets and an online recommender system with over 100 million users. Specifically, UNEX-RL reveals a 0.558% increase in users' usage time compared with single-agent RL algorithms in online A/B experiments, highlighting the effectiveness of UNEX-RL in industrial recommender systems.
BibTeX
@inproceedings{zhang2024unex,
  author = {Gengrui Zhang and Yao Wang and Xiaoshuang Chen and Hongyi Qian and Kaiqiao Zhan and Ben Wang},
  title = {{UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i8.28783},
}
Figure