Recurrent Neural Networks for Stochastic Control in Real-Time Bidding

Nicolas Grislain, Nicolas Perrin, Antoine ThabaultKDD 2019 · DOI · arXiv

In the high-stakes environment of Real-Time Bidding (RTB), advertisers compete in millisecond-long auctions to purchase ad impressions. Managing these campaigns effectively requires balancing a fixed budget against a delivery goal (e.g., a specific number of impressions) while navigating a highly volatile market. Traditional approaches often rely on simple feedback loops, such as Proportional-Integral (PI) controllers, which adjust bids based on current pacing. However, these reactive systems fail to account for market risk—the possibility that prices or volumes might shift unpredictably.

Researchers from Verizon Media, in a paper presented at KDD 2019, propose a novel solution by formulating RTB as a stochastic control problem. By using Recurrent Neural Networks (RNNs) to learn optimal bidding strategies, they demonstrate a significant improvement in cost efficiency and robustness compared to traditional methods.

The Stochastic Control Framework

The core of the paper's approach is the formalization of bidding through the Hamilton-Jacobi-Bellman (HJB) equation, a partial differential equation used to determine optimal strategies in dynamic, uncertain systems. The authors prove that in an uncertain market, the optimal bid should equal the marginal expected cost of reaching the campaign goal.

Because solving the HJB equation in real-time for millions of concurrent campaigns is computationally prohibitive, the authors employ a Gated Recurrent Unit (GRU) to approximate the solution. This RNN-based controller takes inputs such as remaining time, remaining goal, and recent market volume/spend to determine the bid level.

The RNN-based controller architecture uses a hidden state $\hat{H}$ to track market conditions and generate optimal bidding actions $a(\hat{H})$.

As shown in the architecture above, the RNN maintains a 16-dimensional hidden state that encodes a "memory" of market conditions. This allows the system to infer the underlying state of the market directly from data feedback ($S, V$) rather than relying on manually engineered features.

Robustness to Market Shocks

A key advantage of the RNN approach is its ability to anticipate and react to market volatility. In simulations where the market experienced a "shock"—such as a sudden 40% drop in available impressions—the RNN-based bidder adjusted its strategy far more effectively than deterministic models.

Simulation of a bidder's response to a 40% drop in market volume at the end of the day. The orange line represents the bid level adjusting to ensure the goal (blue line) is met.

When the system senses high uncertainty or a sudden drop in volume, it naturally learns to "front-load" delivery. By bidding more aggressively early in the day, the controller effectively purchases insurance against the risk of a late-day market crash. This behavior minimizes the combined cost of ad spend and the financial penalties associated with under-delivery.

Key Results and Production Impact

The model was evaluated using eight days of production logs from a major global Supply Side Platform (SSP), covering approximately 115 million won impressions. The results were significant:

  • Cost Efficiency: The RNN model reduced average delivery costs by approximately 20% compared to standard PI controllers across various goal sizes.
  • Graceful Failure: The system naturally learns the constraint of marginal cost bidding; it will never bid more than the penalty for missing an impression, choosing to pay the penalty if the market becomes too expensive.
  • Scalability: Despite its performance, the model is remarkably lightweight, making it suitable for high-throughput production environments where decisions must be made every few minutes.

Conclusion

By bridging the gap between formal control theory and deep learning, this research demonstrates that RNNs can effectively manage the stochastic risks inherent in programmatic advertising. While future work remains to adapt these models for first-price auctions and multi-objective optimization, the current findings provide a robust foundation for the next generation of real-time bidding controllers.

Summary updated

BibTeX
@inproceedings{grislain2019recurrent,
  author = {Nicolas Grislain and Nicolas Perrin and Antoine Thabault},
  title = {{Recurrent Neural Networks for Stochastic Control in Real-Time Bidding}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2019},
  doi = {10.1145/3292500.3330749},
}
Figure