Abstract
Industrial recommender systems typically employ multi-stage pipelines—matching, ranking, and re-ranking—to process massive item pools. While Reinforcement Learning (RL) excels at optimizing long-term user engagement, its application in these sequential funnels is hindered by complex inter-stage dependencies. We present UNEX-RL, a multi-agent RL framework that models each stage as a cooperative agent. By leveraging a Cascade Information Chain (CIC) to handle unidirectional execution, UNEX-RL addresses the challenges of observation dependency and cascading effects. Evaluation on the KuaiRand dataset and a large-scale production environment shows significant improvements in user watch time and retention.
The Multi-Stage Recommendation Challenge
Modern industrial recommender systems are structured as "funnels" to balance efficiency and accuracy. A typical pipeline consists of Matching (retrieving thousands of candidates), Ranking (scoring hundreds), and Re-ranking (final ordering).
While RL is ideal for optimizing long-term rewards like total watch time, applying it to these multi-stage systems is difficult. Standard RL treats the entire system as a single "black box," ignoring stage-specific constraints. Conversely, standard Multi-Agent RL (MARL) assumes simultaneous actions, whereas recommendation funnels exhibit unidirectional execution: the output of an upstream stage directly determines the input of the downstream stage. This leads to two primary hurdles:
- Observation Dependency (OD): Changes in an early stage's strategy render historical training data for later stages obsolete.
- Cascading Effect (CE): Actions in early stages ripple through the pipeline, but standard training fails to account for these downstream impacts.
The UNEX-RL Framework
UNEX-RL (UNidirectional-EXecution-based RL) treats each stage as a cooperative agent within a unified framework. The core innovation is the Cascade Information Chain (CIC), which ensures that the system accounts for the sequential nature of decision-making.
Cascade Information Chain (CIC)
Instead of training on static snapshots from a replay buffer, CIC "replays" the recommendation process during training. It starts with the initial user state and iteratively calculates observations and actions for every subsequent stage. This ensures the critic evaluates performance based on a consistent, updated view of inter-stage interactions, effectively mitigating OD and CE.
Variance Reduction Techniques
To handle the high noise in user behavior data, UNEX-RL introduces:
- Stopping Gradient (SG): This stabilizes gradients by "cutting" certain paths, focusing on the most direct impact of an agent's action.
- Category Quantile Rescale (CQR): This normalizes rewards by comparing them to similar users and items (quantiles) rather than using raw values like watch time, providing a cleaner learning signal.
Experimental Results
The framework was evaluated on the KuaiRand public dataset and through live A/B testing on the Kuaishou short-video app, which serves over 100 million daily active users.
- Offline Performance: On KuaiRand, UNEX-RL achieved a WatchTime of 1056.2s, significantly outperforming single-agent DDPG (732.6s) and traditional multi-agent approaches (782.5s).
- Online Impact: In production, UNEX-RL delivered a 0.558% increase in total user usage time compared to state-of-the-art single-agent RL.
- Ablation Study: Removing the CIC component led to a sharp decline in performance, confirming that managing the cascading effect is vital for multi-stage optimization.
Conclusion
UNEX-RL demonstrates that treating a recommendation funnel as a cooperative multi-agent system with unidirectional execution is superior to black-box or independent agent approaches. By explicitly modeling the dependencies between stages via the Cascade Information Chain, UNEX-RL unlocks significant gains in long-term user engagement for large-scale industrial platforms.
Summary updated