Introduction
In modern e-commerce and food delivery platforms, the user experience is defined by the arrangement of items in a feed. Platforms must balance the placement of advertisements (which generate direct revenue) with organic items (which drive long-term user retention and service fees). This "ads allocation" problem has evolved from simple point-wise models, which evaluate items in isolation, to list-wise models that consider the entire arrangement of a screen.
While list-wise models capture the complex interactions between items, they suffer from the "curse of dimensionality." With millions of items and dozens of slots, the number of possible configurations is astronomical. This makes it difficult for standard Reinforcement Learning (RL) agents to learn generalizable representations, often resulting in poor sample efficiency and suboptimal performance. This paper proposes a unified RL framework that overcomes these challenges by incorporating three domain-specific auxiliary tasks to guide the learning of list-wise representations.
The Proposed Framework
The core of the proposed approach is to augment a base RL agent (such as CrossDQN) with auxiliary signals that force the model to learn more meaningful latent representations of the state-action space. The authors introduce three specific tasks:
1. Reconstruction-based Auxiliary Task (RAT)
The RAT ensures that the latent representation does not lose critical raw information. Using an encoder-decoder architecture, the model attempts to reconstruct essential binary features of the items in the list, such as whether a merchant is a "brand" partner or if the delivery is free. By forcing the model to reconstruct these features, the authors ensure the representation remains grounded in the attributes that most influence user decision-making.
2. Prediction-based Auxiliary Task (PAT)
RL agents often struggle with sparse rewards, such as an actual purchase or order. To provide a denser feedback signal, the PAT requires the model to predict intermediate user behaviors: the Click-Through Rate (CTR) for each slot and the probability of a "pull-down" event (indicating whether a user continues to browse or leaves the session). This supervised signal stabilizes the early stages of training and helps the model understand the immediate impact of an allocation.
3. Contrastive-Learning based Auxiliary Task (CLAT)
To help the model distinguish between different arrangements, the authors employ contrastive learning. They introduce a clever "unexposed item" heuristic: if a user only scrolls to the 7th slot, the items in slots 8 through 10 have no impact on their behavior. By replacing these unexposed items with random ones, the authors create "positive" samples. The model is trained to recognize that the underlying state-action value should remain similar despite these changes, improving the robustness of the representation.
Experimental Evaluation
Dataset and Setup
The framework was evaluated using a massive real-world dataset from the Meituan food delivery platform, comprising over 12.7 million requests and 2 million users. The inventory included approximately 385,000 ads and 726,000 organic items. To ensure a diverse range of state-action pairs for offline training, the data was collected using a random exploratory policy.
Performance Results
The proposed method was compared against several state-of-the-art baselines, including HRL-Rec, DEAR, and CrossDQN.
Offline Evaluation: - Ads Revenue ($R^{ad}$): Increased by 2.38% over the best-performing baseline. - Service Fee ($R^{fee}$): Increased by 1.81%.
Online A/B Testing: Following successful offline results, the model was deployed on the Meituan platform for a live A/B test, where it demonstrated significant real-world impact:
- Ads Revenue: +2.92%
- Order Rate ($R^{cxr}$): +2.21%
- User Experience Score: +1.13%
Ablation studies further revealed that all three auxiliary tasks are vital; removing any single task resulted in a measurable decline in performance, confirming that reconstruction, prediction, and contrastive learning provide complementary signals.
Discussion and Limitations
While the framework provides a significant boost in performance, it introduces new complexities. The model relies on three hyperparameters to balance the weights of the auxiliary tasks, currently requiring a computationally expensive grid search to optimize. Additionally, like many offline RL approaches, the model must contend with "distribution shift," where the policy may encounter unfamiliar states in a live environment that were not present in the training data. Finally, the inclusion of multiple auxiliary MLPs increases the training overhead, though the authors suggest this can be managed by controlling the size of the contrastive sample sets.
Conclusion
By integrating domain-specific auxiliary tasks into a list-wise RL framework, this research demonstrates a path forward for high-dimensional optimization in e-commerce. The use of reconstruction, CTR prediction, and a novel contrastive learning heuristic allows the agent to learn efficient representations that translate directly into increased revenue and improved user satisfaction on a major industrial platform.
Summary updated