Abstract
In recent years, reinforcement learning (RL) has emerged as a powerful tool for optimizing long-term rewards in recommender systems. However, industrial systems are typically multi-stage pipelines (e.g., matching, ranking, re-ranking), presenting a challenge for single-agent RL due to varying observation spaces across stages. We propose UNEX-RL, a novel UNidirectional-EXecution-based multi-agent RL framework. We identify two primary challenges in this setting: observation dependency and the cascading effect. To address these, we introduce the Cascading Information Chain (CIC) method to effectively train the system by separating independent and action-dependent observations. Evaluated on public datasets and a production system with over 100 million users, UNEX-RL demonstrated a 0.558% increase in user usage time compared to state-of-the-art single-agent RL methods.
Introduction
Modern industrial recommender systems are not monolithic; they are complex, multi-stage pipelines designed to filter millions of items down to a handful of recommendations. While traditional approaches focus on immediate feedback like clicks, there is a growing shift toward optimizing long-term user engagement metrics, such as session length and retention. While Reinforcement Learning (RL) is naturally suited for such long-term optimization, its application in multi-stage environments is non-trivial. Standard Multi-Agent RL (MARL) often assumes agents act simultaneously, but recommendation pipelines are inherently sequential and unidirectional.
The Challenge of Multi-Stage Systems
In a typical recommendation pipeline, the output of one stage (e.g., the "Matching" stage) becomes the input for the next (e.g., the "Ranking" stage). This funnel-like structure creates a unique environment for RL agents.
This unidirectional execution introduces two major hurdles:
- Observation Dependency (OD): Downstream observations are not static; they depend entirely on the actions taken by upstream agents. This makes standard offline training from fixed replay buffers inconsistent.
- Cascading Effect (CE): An upstream agent's action affects the policy and performance of all subsequent agents. Ignoring this interaction leads to sub-optimal global performance, as agents may work at cross-purposes.
The UNEX-RL Framework
To overcome these challenges, we propose the UNEX-RL framework. UNEX-RL treats each stage of the recommendation pipeline as an independent agent within a cooperative MARL setup. The core of the framework is the Cascading Information Chain (CIC).
Cascading Information Chain (CIC)
Instead of sampling independent transitions from a buffer, CIC "replays" the entire execution sequence during training. Starting from the initial user state, it iteratively calculates the observations and actions for every stage in the pipeline. This ensures that the critic—the component evaluating the actions—receives a consistent and accurate flow of information that accounts for the cascading dependencies between stages.
Variance Reduction Techniques
Industrial recommendation data is notoriously noisy. To stabilize training, we employ two key techniques:
- Category Quantile Rescale (CQR): This method normalizes rewards (like watch time) by calculating their quantile relative to similar users and items. This strips away inherent user bias (e.g., some users naturally watch more videos) to reveal the true reinforcement signal.
- Stopping Gradient (SG): To prevent complex cascading gradients from becoming unstable during backpropagation, we focus the learning on the direct impact of an agent's action on the global reward.
Experimental Results
We evaluated UNEX-RL using the KuaiRand public dataset and live A/B testing on the Kwai App, a short-video platform with over 100 million daily active users.
- Offline Performance: On KuaiRand, UNEX-RL-CIC consistently outperformed single-agent baselines (DDPG, TD3) and standard MARL approaches across metrics like total WatchTime and Session Length.
- Online Impact: In production A/B tests, UNEX-RL-CIC achieved a 0.953% increase in daily watch time over the baseline and a 0.558% increase over the previous state-of-the-art single-agent RL approach. In large-scale industrial systems, gains of this magnitude are highly significant.
Conclusion
UNEX-RL demonstrates that the sequential, unidirectional nature of recommendation pipelines is a critical factor that must be modeled to achieve optimal long-term rewards. By using the Cascading Information Chain to manage observation dependencies and cascading effects, UNEX-RL enables effective multi-agent cooperation. Future work will look into more complex stage interactions and broader reward signals, such as multi-day user retention and ecosystem health.
Summary updated