Comparison Lift: Bandit-based Experimentation System for Online Advertising

Tong Geng, Xiliang Lin, Harikesh S. Nair, Jun Hao, Bin Xiang, Shurui Fan

Abstract

Comparison Lift is an experimentation-as-a-service (EaaS) application for testing online advertising audiences and creatives at JD.com. Unlike many other EaaS tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. The advantages of the bandit-based approach are two-fold. First, it aligns the randomization induced in the test with the advertiser’s goals from testing. Second, by adapting experimental design to information acquired during the test, it reduces substantially the cost of experimentation to the advertiser. Since launch in May 2019, Comparison Lift has been utilized in over 1,500 experiments. We estimate that utilization of the product has helped increase click-through rates of participating advertising campaigns by 46% on average. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design. Both suggest significant value generation and cost savings to advertisers from the product.

Introduction

In the high-stakes environment of online advertising, selecting the right combination of creative assets (images, videos, text) and target audiences is critical. Traditional experimentation typically relies on fixed-sample A/B/n testing. While statistically robust, these methods incur significant opportunity costs by allocating equal traffic to underperforming variants throughout the test duration. Furthermore, digital marketing audiences often overlap, complicating the process of obtaining unbiased performance estimates.

To address these challenges, JD.com developed Comparison Lift (CL), an experimentation-as-a-service platform that replaces static testing with adaptive contextual bandits. By dynamically shifting traffic toward winning combinations in real-time, CL maximizes advertiser payoff while simultaneously learning optimal campaign configurations.

Methodology

The Disjoint Audience Framework

To resolve the issue of overlapping target audiences, CL partitions all targeted users into a set of Disjoint Audiences (DAs). Each user is mapped to exactly one DA, which serves as the "context" for the bandit algorithm. The system learns the optimal creative for these non-overlapping sub-populations and aggregates the findings back to the original target audiences, ensuring unbiased estimation.

Adaptive Allocation via Thompson Sampling

CL utilizes a Bayesian framework, specifically a Beta-Bernoulli model, to estimate the click-through rate (CTR) for each creative-audience pair. The system employs a Contextual Thompson Sampling (TS) algorithm to manage the exploration-exploitation trade-off. As data is collected, the algorithm calculates the posterior probability that a specific combination is the "best arm" and adjusts traffic allocation accordingly.

Probabilistic graphical model of the Comparison Lift bandit system showing the transition of latent variables and decisions over time.

This approach aligns the randomization process with the advertiser's primary goal: maximizing clicks. Unlike frequentist p-values, the Bayesian framework allows advertisers to monitor progress and stop tests at any time without compromising statistical validity.

Implementation and Scale

The system was deployed at JD.com and utilized in over 1,500 experiments between May 2019 and June 2020. A typical experiment involves testing approximately 2.4 creatives across 3.8 target audiences over a 5-day period. To maintain low latency in the ad-serving pipeline, the system currently limits experiments to 25 unique combinations.

Key Results

Performance Gains

The adoption of Comparison Lift led to a 46% average increase in CTR for the best-performing options discovered by the tool compared to baseline advertiser selections. More importantly, the adaptive design reduced the cost of experimentation itself, generating 27% more clicks during the testing phase than a simulated fixed-sample A/B design.

Case Study: Algorithm Trajectory

A detailed analysis of a Samsung campaign—involving two target audiences and three creatives (six total combinations)—illustrates the algorithm's efficiency. As shown in the figure below, the system rapidly identifies the superior combination (C-TA 3), shifting the majority of traffic to it within approximately 25 batches.

Convergence of posterior probabilities for six creative-audience combinations in a Samsung campaign case study.

By the conclusion of a typical test, the average probability of correctly identifying the "best" arm reached 73.7%, even in cases where advertisers chose to terminate the experiment early.

Conclusion and Future Work

Comparison Lift demonstrates that bandit-based experimentation can significantly outperform traditional A/B testing in production advertising environments. Future developments will focus on optimizing for sparser, delayed signals like conversions and total revenue, rather than just immediate CTR. Additionally, JD.com aims to extend this framework to external programmatic environments, which will require accounting for more complex auction dynamics.

Summary updated

Abstract

Comparison Lift is an experimentation-as-a-service (EaaS) application for testing online advertising audiences and creatives at JD.com. Unlike many other EaaS tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. The advantages of the bandit-based approach are two-fold. First, it aligns the randomization induced in the test with the advertiser’s goals from testing. Second, by adapting experimental design to information acquired during the test, it reduces substantially the cost of experimentation to the advertiser. Since launch in May 2019, Comparison Lift has been utilized in over 1,500 experiments. We estimate that utilization of the product has helped increase click-through rates of participating advertising campaigns by 46% on average. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design. Both suggest significant value generation and cost savings to advertisers from the product.
BibTeX
@inproceedings{geng2021comparison,
  author = {Tong Geng and Xiliang Lin and Harikesh S. Nair and Jun Hao and Bin Xiang and Shurui Fan},
  title = {{Comparison Lift: Bandit-based Experimentation System for Online Advertising}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=Vc3IfWhA36s},
}
Figure