LADDER: A Human-Level Bidding Agent for Large-Scale Real-Time Online Auctions

Yu Wang, Jiayi Liu, Yuxiang Liu, Jun Hao, Yang He, Jinghe Hu, Weipeng P. Yan, Mantian LiarXiv 2017 · arXiv

Abstract

Real-Time Bidding (RTB) in online advertising is a high-stakes environment characterized by massive state spaces and millisecond latency requirements. Traditional bidding strategies often rely on heavy feature engineering or manual expert tuning. This paper introduces LADDER (Learning Ad Exchange Bidder), a deep reinforcement learning agent that learns directly from high-level semantic text. By employing a novel Deep Asynchronous Stochastic Q-Network (DASQN) architecture, LADDER decouples the acting, observing, and training processes, enabling real-time performance. Experimental results from JD.com demonstrate that LADDER significantly outperforms human-engineered baselines, achieving over 50% growth in ad revenue and a 17% increase in advertiser ROI.

The Complexity of Real-Time Bidding

Real-Time Bidding involves Demand Side Platforms (DSPs) competing in automated auctions to display ads to users. This environment is an "auction game of incomplete information" with astronomical solution spaces ($10^{10^7}$). Existing systems face significant challenges, including extreme latency (millisecond response times), high stochasticity in user behavior, and the burden of updating billions of features in real-time to understand semantic data like product IDs.

The Real-Time Bidding (RTB) ecosystem involving publishers, SSPs, and competing DSPs.

The LADDER Methodology

LADDER models the auction as a Partially Observable Markov Decision Process (POMDP). Its success is driven by two core innovations:

  • DASQN Architecture: To handle high-throughput production environments (100,000+ auctions/sec), the Deep Asynchronous Stochastic Q-Network (DASQN) fully decouples the serving, observing, and training loops. This allows the agent to make bidding decisions in milliseconds while learning from delayed rewards in the background.
  • Semantic Text Processing: Rather than relying on massive embedding tables for product IDs (SKUs), LADDER treats IDs as plain text. A character-level Convolutional Neural Network (CNN) processes these strings, allowing the model to recognize patterns in product categories and numbering rules without extensive feature engineering.
  • GSP-Aware Data Augmentation: Utilizing the logic of Generalized Second Price (GSP) auctions, the authors augment training data by recognizing that any bid higher than the winning price would have also secured the auction, improving learning efficiency.

Key Results and Performance

Evaluated on real-world production data from JD.com, LADDER demonstrated superior performance during major sales events compared to JD's previous state-of-the-art "human-machine hybrid" system.

Performance comparison of LADDER against baselines, showing significant gains in reward, revenue, and CTR growth.

The model achieved a 50% increase in ad revenue and a 35% improvement in Click-Through Rate (CTR). Furthermore, advertiser ROI improved by 17%, indicating that the agent discovered efficient bidding strategies that benefit both the platform and the advertisers. Visualization of the model's latent space showed that a single LADDER model successfully learned to distinguish between different publishers and complex user behaviors without explicit instruction.

Conclusion and Future Work

LADDER represents a significant step forward in applying deep reinforcement learning to large-scale industrial problems. By treating IDs as language and utilizing an asynchronous training architecture, it overcomes the traditional barriers of scale and latency in RTB. Future work aims to extend this RL framework to the ad ranking phase to further optimize the advertising pipeline and explore its application in other high-throughput domains like quantitative trading.

Summary updated

BibTeX
@misc{wang2017ladder,
  author = {Yu Wang and Jiayi Liu and Yuxiang Liu and Jun Hao and Yang He and Jinghe Hu and Weipeng P. Yan and Mantian Li},
  title = {{LADDER: A Human-Level Bidding Agent for Large-Scale Real-Time Online Auctions}},
  year = {2017},
  eprint = {1708.05565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.05565},
}
Figure