Abstract
Traditional ad auctions in large-scale recommender systems are typically "myopic," optimizing for immediate engagement metrics like click-through rates (eCTR) or conversion rates (eCVR). This approach often ignores the downstream effects of current recommendations on long-term user retention and platform health. This paper introduces a framework that integrates Reinforcement Learning (RL) into the existing auction mechanism using a one-step policy improvement approach. By augmenting the standard auction score with a SARSA-based long-term value (LTV) term, the system biases selections toward actions that benefit the user over time. Deployed on Meta’s production systems, this method achieved a 4–10% lift in conversions in a massive A/B test involving 80 million users, demonstrating that long-term optimization can be achieved without discarding established auction frameworks.
Introduction: The Myopia of Modern Ad Auctions
Most online advertising platforms, including Meta, Google, and Yahoo, utilize auction-based mechanisms to determine which content to display to a user. These auctions traditionally calculate a score for each candidate ad based on the advertiser's bid and the estimated probability of an immediate action (e.g., $Bid \times eCTR$). While effective for short-term revenue and relevance, this "myopic" strategy fails to account for the sequential nature of user interaction.
A high-click but low-quality ad might generate immediate revenue but degrade the user experience, leading to lower platform return rates in the future. Conversely, showing a highly relevant but lower-bidding ad today might increase the probability of the user returning tomorrow. The challenge lies in transitioning these complex, multi-billion-user systems toward long-term value (LTV) without disrupting the fairness and business logic inherent in the auction framework.
Methodology: One-Step Policy Improvement
Rather than replacing the entire auction system with a pure RL agent—a move that would be computationally prohibitive and risky for business stability—the authors propose a "hybrid" approach. They treat the problem as a Markov Decision Process (MDP) and apply a one-step policy improvement.
The RL Framework
In this MDP, the state represents the user's historical interactions and context, the action is the selection of a specific ad or content item, and the reward is defined as a binary conversion (a proxy for high-value engagement). To estimate the value of these actions, the researchers utilize SARSA (State-Action-Reward-State-Action), an on-policy RL algorithm.
SARSA is used to learn a Q-function, $\hat{Q}_{\pi_{base}}(s, a)$, which predicts the expected cumulative future rewards if the system takes action $a$ in state $s$ and continues to follow the current "base" auction policy thereafter. The choice of an on-policy method is critical; off-policy methods like Q-learning often suffer from instability and "distribution shift" in massive production environments where the agent might attempt to learn from actions the current system would never realistically take.
The Hybrid Scoring Mechanism
The core innovation is the modification of the auction selection criteria. Instead of relying solely on the traditional auction score $f(s, a)$, the system uses a weighted combination:
$$\text{Score} = (1 - \alpha) \cdot f(s, a) + \alpha \cdot \hat{Q}_{\pi_{base}}(s, a)$$
Here, $\alpha$ is a hyperparameter that controls the influence of the RL component. In production, $\alpha$ is tuned to ensure the RL bias contributes a controlled portion of the total score (typically less than 8%), allowing for a safe and measurable transition toward long-term optimization.
Theoretical Grounding
The authors provide a theoretical guarantee for this approach based on dynamic programming. They prove that by adding the estimated Q-value of the current policy to the scoring function, the resulting modified policy is guaranteed to perform at least as well as—and usually better than—the base policy in terms of cumulative long-term rewards. This "one-step" improvement provides a mathematically sound shortcut to optimization without requiring the system to reach a fully converged optimal RL policy.
Experimental Setup and Scale
The system was tested within Meta’s production recommender infrastructure, which manages billions of impressions daily.
- A/B Testing: A six-week online experiment was conducted with 40 million users in the test group and 40 million in the control group.
- Data Management: The researchers utilized a replay buffer containing approximately 300 million data tuples, updated daily to account for rapid shifts in user behavior and content trends.
- Horizon: The current implementation uses an effective time horizon of 15 days for calculating cumulative rewards.
Key Results
The results of the large-scale A/B test confirmed the efficacy of the RL-biased auction:
- Significant Conversion Lift: The test group saw a 4% to 10% increase in both the total number of conversions and the conversion rate compared to the standard myopic auction.
- Stability: These gains were not fleeting; the performance improvements remained stable throughout the entire six-week testing period.
- Impression Neutrality: Crucially, the number of daily impressions remained nearly identical between the test and control groups. This indicates that the system achieved higher value by showing better ads, rather than simply increasing the volume of ads shown to users.
Discussion and Future Work
This research demonstrates that "evolutionary" changes to auction systems—nudging them with RL—can yield "revolutionary" results in user value. By treating the RL output as a bias term, platforms can maintain the integrity of their auction logic while optimizing for the long game.
Future work aims to extend the effective horizon from 15 days to 60 days to capture even more subtle long-term user trends. Additionally, the researchers are exploring more complex reward functions that move beyond binary conversions to include multi-faceted metrics like user sentiment and social network effects, where one user's positive experience influences the retention of others.
Summary updated