Abstract
Modern listwise recommendation systems need to consider both long-term user perceptions and short-term interest shifts. Reinforcement learning can be applied on recommendation to study such a problem but is also subject to large search space, sparse user feedback and long interactive latency. Motivated by recent progress in hierarchical reinforcement learning, we propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation. Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy by modeling the process as a sequential decision-making problem. We argue that such framework has a well-defined decomposition of the outra-session context and the intra-session context, which are encoded by the high-level and low-level agents, respectively. To verify this argument, we implement both a simulator-based environment and an industrial dataset-based experiment. Results observe significant performance improvement by our method, compared with several well-known baselines. Data and codes have been made public.
Introduction
Modern recommendation systems face a "Goldilocks" problem: they must be broad enough to understand long-term user shifts across multiple sessions while remaining fast enough to react to immediate, item-by-item interactions. Standard Reinforcement Learning (RL) approaches often struggle with this balance. They either treat an entire session as a single action, ignoring intra-session interest shifts, or they model every item selection as a step in a massive sequence, leading to the "Curse of Dimensionality" and sparse rewards.
Furthermore, rich user features—such as precise location or dwell time—are often restricted to mobile devices due to privacy concerns or the high cost of real-time cloud uploading. To address these challenges, we introduce mccHRL, a mobile-cloud collaborative Hierarchical Reinforcement Learning framework.
The mccHRL Architecture
The mccHRL framework decouples recommendation into two layers of decision-making. The High-Level recommendation Agent (HRA) resides on the cloud and operates on a "slow" time scale (session-to-session). It analyzes "outra-session" context, such as long-term preferences and spatiotemporal status, to set a high-level goal—an embedding representing the ideal session characteristics.
The Low-Level recommendation Agent (LRA) resides on the mobile device (edge) and operates on a "fast" time scale (item-to-item). It uses the goal provided by the HRA alongside real-time on-device features to select the optimal next item. This collaborative structure allows the cloud to handle heavy historical analysis while the device manages reflexive, immediate item selection.
Hierarchical Temporal Abstraction
A core innovation of mccHRL is its use of temporal abstraction to solve the sparse reward problem. In traditional RL, an agent might only receive a reward after a user clicks an item at the end of a list. In our hierarchical model, the LRA receives "intrinsic" rewards based on how well its choices align with the HRA's goal, even before final user feedback is received.
This "fast-slow" paradigm ensures that the system can adapt to mid-session changes in user behavior without requiring constant communication with the cloud. By moving item-selection logic to the device, we significantly reduce interactive latency and better utilize privacy-sensitive on-device data.
Experimental Results and Impact
We evaluated mccHRL using both a MovieLens-100k simulator and a massive industrial dataset from Alibaba (Taobao’s front-page feed). The results demonstrate significant performance gains:
- Accuracy: On the Alibaba dataset, mccHRL achieved a D-AUC of 0.853, outperforming the standard Deep Interest Network (DIN) which scored 0.781.
- Ablation Studies: Removing edge features or the LRA logic led to noticeable performance drops, confirming that both the hierarchy and on-device processing are critical.
- Real-World Deployment: In a week-long A/B test on a world-leading mobile app, mccHRL increased the global Click-Through Rate (CTR) by 1.2% and total views by 1.5%.
Conclusion
The mccHRL framework represents a significant step forward in listwise recommendation by effectively combining cloud-based long-term modeling with edge-based real-time adaptation. By leveraging hierarchical reinforcement learning and temporal abstraction, the system overcomes traditional RL limitations regarding search space and reward sparsity. Future work will focus on incorporating instant, non-click user feedback (e.g., scrolling behavior) directly into the LRA's reward structure for even more dynamic reranking.
Summary updated