Comparison Lift: Bandit-based Experimentation System for Online Advertising

Tong Geng, Xiliang Lin, Harikesh S. Nair, Jun Hao, Bin Xiang, Shurui Fan

Abstract

Comparison Lift is an experimentation-as-a-service (EaaS) application for testing online advertising audiences and creatives at JD.com. Unlike many other EaaS tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. The advantages of the bandit-based approach are two-fold. First, it aligns the randomization induced in the test with the advertiser's goals from testing. Second, by adapting experimental design to information acquired during the test, it reduces substantially the cost of experimentation to the advertiser. Since launch in May 2019, Comparison Lift has been utilized in over 1,500 experiments. We estimate that utilization of the product has helped increase click-through rates of participating advertising campaigns by 46% on average. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design. Both suggest significant value generation and cost savings to advertisers from the product.

Introduction

In digital marketing, advertisers face a combinatorial explosion of choices regarding which creative (image, video, or text) works best for specific target audiences. Traditional "A/B/n" or split tests suffer from two major flaws in this environment: the "cost of curiosity," where traffic is wasted on poor-performing variants to reach statistical significance, and the "overlap problem," where target audiences are not mutually exclusive, complicating statistical inference.

Comparison Lift addresses these challenges by replacing static A/B testing with a contextual bandit framework. This system dynamically shifts traffic toward high-performing ad-audience combinations, optimizing for the advertiser's expected payoff rather than just statistical learning. Since its deployment at JD.com, the system has demonstrated significant improvements in campaign efficiency and advertiser ROI.

Methodology

The system implements a Contextual Thompson Sampling (TS) algorithm within a Bayesian framework. To handle overlapping target audiences, Comparison Lift utilizes Disjoint Audience (DA) Splitting, slicing overlapping segments into mutually exclusive sub-populations. These segments serve as "contexts" for the bandit, allowing it to learn the optimal creative for each specific slice of the population.

System architecture of Comparison Lift, illustrating the integration between the retrieval server, ad server, and real-time data aggregation.

The technical architecture integrates directly with JD.com's ad delivery pipeline. Bandits are triggered at the Retrieval Server, and model parameters are updated in batches using aggregated impression and click data from a real-time aggregator. This loop allows for adaptive traffic allocation: as data accumulates, the algorithm automatically funnels more traffic to winning combinations.

Results and Evaluation

The system's performance was evaluated across 1,547 live experiments conducted between May 2019 and June 2020. Key findings include:

  • CTR Improvement: Campaigns utilizing the best-discovered options saw a 46% average increase in CTR compared to a no-testing counterfactual.
  • Reduced Opportunity Cost: The adaptive design generated 27% more clicks during the testing phase itself compared to standard fixed-sample A/B designs.
Posterior probability of being the best combination over time, demonstrating rapid convergence to the optimal ad-audience pair.

A case study involving a Samsung campaign highlights the system's speed. As shown in the convergence plot, the system rapidly identifies the optimal creative-audience combination. In this instance, the system reached a 99.91% posterior probability for the winner, which achieved a CTR 97% higher than the worst-performing variant.

Conclusion and Future Work

Comparison Lift demonstrates that bandit-based designs can significantly lower the barrier to experimentation for advertisers by reducing opportunity costs. However, several challenges remain. The system currently caps creative-audience combinations at 25 to manage latency and statistical power. Furthermore, while the system optimizes for CTR, future iterations aim to address more complex metrics like conversions or revenue, which involve rarer events and delayed feedback. Finally, accounting for dynamic auction pressures remains a critical area for improving causal inference in production environments.

Summary updated

Abstract

Comparison Lift is an experimentation-as-a-service (EaaS) application for testing online advertising audiences and creatives at JD.com. Unlike many other EaaS tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. The advantages of the bandit-based approach are two-fold. First, it aligns the randomization induced in the test with the advertiser's goals from testing. Second, by adapting experimental design to information acquired during the test, it reduces substantially the cost of experimentation to the advertiser. Since launch in May 2019, Comparison Lift has been utilized in over 1,500 experiments. We estimate that utilization of the product has helped increase click-through rates of participating advertising campaigns by 46% on average. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design. Both suggest significant value generation and cost savings to advertisers from the product.
BibTeX
@inproceedings{geng2021comparison,
  author = {Tong Geng and Xiliang Lin and Harikesh S. Nair and Jun Hao and Bin Xiang and Shurui Fan},
  title = {{Comparison Lift: Bandit-based Experimentation System for Online Advertising}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=Vc3IfWhA36s},
}
Figure