Introduction
In modern e-commerce and food delivery platforms, the "feed" serves as the primary interface for user engagement. A critical challenge for these platforms is the "position allocation" problem: determining the optimal mix of content types—such as short-form videos and traditional graphic-text items—at specific positions within a user's scroll. Effective allocation maximizes user engagement (Click-Through Rate, or CTR) and platform revenue (Gross Merchandise Volume, or GMV), while poor allocation leads to user churn.
While Reinforcement Learning (RL) is theoretically well-suited for this sequential decision-making task, its deployment in industrial settings is hindered by significant data bias. This paper introduces MDDL (Multi-Distribution Data Learning), a framework developed by researchers at Meituan and Tsinghua University to optimize multi-channel feeds by effectively leveraging heterogeneous data sources.
The Challenge: Strategy vs. Random Data
Training an RL agent for position allocation typically involves two types of data, each with distinct drawbacks:
- Strategy Data: This consists of massive datasets generated by existing online models. While rich in features, this data is highly biased toward the current model's preferences. Because the model only explores actions it deems favorable, the resulting distribution is imbalanced, leading to Q-value overestimation. The RL agent becomes overconfident in certain actions simply because it lacks exposure to alternatives.
- Random Data: Collected through random exploration, this data is unbiased and covers the action space uniformly. However, it is inherently limited in scale ("small data") because random exploration negatively impacts user experience and revenue, making it expensive to collect.
Standard RL approaches often fail when mixing these sources; the sheer volume of biased Strategy Data tends to overwhelm the signal from the Random Data, causing the model to inherit the biases and overestimations of the previous system.
The MDDL Framework
To address these challenges, the MDDL framework treats the two data sources differently within a unified training loop, employing a hybrid loss function and a novel distance metric.
1. Dual-Source Learning Strategy
Instead of applying a uniform RL objective, MDDL bifurcates its learning process:
- Imitation Learning (IL) for Strategy Data: The framework treats the current online strategy as an "expert." By using an imitation learning signal, the model is encouraged to stay close to the existing strategy when processing biased data, preventing the wild Q-value inflation that occurs when standard RL is applied to biased distributions.
- Reinforcement Learning (RL) for Random Data: Because the random data is unbiased, the model uses the standard Bellman equation to maximize rewards. This allows the agent to learn where it can actually improve upon the current expert strategy.
2. Weighted Exposure Ratio (WER)
A core innovation of MDDL is the Weighted Exposure Ratio (WER). In position allocation, actions are often represented as bit-vectors (e.g., [1, 0, 1] indicating video content at positions 1 and 3). Standard distance metrics like Euclidean distance fail to capture the semantic reality of a user interface—specifically, that a change at the top of the screen is far more significant than a change at the bottom.
WER converts these discrete bit-vectors into a scalar value based on the exposure probability and historical CTR of each position. This provides a "semantic bridge," allowing the model to mathematically quantify how similar two different layouts feel to a user, which is essential for effective imitation learning.
3. Soft Argmax for Differentiability
To ensure the training process remains smooth and differentiable despite the discrete nature of picking UI layouts, the researchers implemented a "soft" version of the argmax function. This allows the gradients to flow through the network more effectively during optimization.
Experimental Results
The researchers evaluated MDDL using real-world logs from the Meituan food delivery platform, comprising approximately 390 million requests from 34 million users. Following offline validation, a week-long online A/B test was conducted on 10% of Meituan's production traffic.
Key Performance Metrics:
- CTR (Click-Through Rate): Increased by 5.46%.
- GMV (Gross Merchandise Volume): Increased by 5.83%.
- Stability: In offline tests, MDDL demonstrated superior stability, with a "Standard Deviation of Overestimation Degree" of 1.228, significantly lower than standard RL approaches which reached as high as 4.417.
Conclusion and Future Work
MDDL provides a robust solution for the position allocation problem by successfully mitigating the overestimation trap inherent in biased industrial data. By combining imitation learning on large-scale biased data with reinforcement learning on small-scale unbiased data, the framework achieves significant gains in both engagement and revenue.
Future research directions include refining hyperparameter sensitivity—specifically the balance between imitation and RL losses—and scaling the framework to handle more complex multi-channel environments involving ads, live streams, and diverse interactive components.
Summary updated