Abstract
Firms implementing digital advertising campaigns face a complex problem in determining the right match between their advertising creatives and target audiences. Typical solutions to the problem have leveraged non-experimental methods, or used "split-testing" strategies that have not explicitly addressed the complexities induced by targeted audiences that can potentially overlap with one another. This paper presents an adaptive algorithm that addresses the problem via online experimentation. The algorithm is set up as a contextual bandit and addresses the overlap issue by partitioning the target audiences into disjoint, non-overlapping sub-populations. It learns an optimal creative display policy in the disjoint space, while assessing in parallel which creative has the best match in the space of possibly overlapping target audiences. Experiments show that the proposed method is more efficient compared to naive "split-testing" or non-adaptive "A/B/n" testing based methods. We also describe a testing product we built that uses the algorithm. The product is currently deployed on the advertising platform of JD.com, an eCommerce company and a publisher of digital ads in China.
Introduction
In the digital advertising ecosystem, the effectiveness of a campaign hinges on finding the optimal match between an ad "creative" and a specific "target audience" (TA). Traditionally, advertisers have relied on "split-testing" or A/B/n testing to evaluate these matches. However, these methods face significant hurdles in real-world environments where target audiences are rarely mutually exclusive.
For instance, an advertiser might target "Male users" and "San Francisco residents." A user who is both male and lives in San Francisco falls into both categories, creating an assignment dilemma: which experimental arm should they be assigned to? Assigning them to only one group biases the results of the other, while excluding them entirely wastes valuable data. Furthermore, traditional static testing allocates traffic equally throughout the experiment, incurring a high "opportunity cost" by continuing to show low-performing ads long after their inferiority is statistically evident.
The Proposed Framework: Contextual Thompson Sampling
To address these challenges, this research introduces an adaptive contextual bandit algorithm designed to handle overlapping audiences through a three-step process:
1. Disjoint Partitioning
The core innovation lies in transforming overlapping Target Audiences (TAs) into a set of mutually exclusive Disjoint Audiences (DAs). For $K$ target audiences, the algorithm creates up to $2^K - 1$ distinct buckets. In the example of "Males" and "SF Residents," the system creates three disjoint buckets: "Males NOT in SF," "SF residents NOT male," and "Males IN SF." This ensures that every user arriving at the platform maps to exactly one context.
2. Contextual Bandit Formulation
The system treats each DA as a "context" and each ad creative as an "arm." It employs Thompson Sampling (TS), a Bayesian approach that adaptively shifts traffic toward the creatives that demonstrate higher performance within each specific sub-population. This allows the system to maximize clicks (or other objectives) even while the experiment is still running.
3. Probabilistic Aggregation and Stopping
While the algorithm learns at the granular DA level, advertisers typically need answers at the aggregate TA level. The framework uses the law of total probability to aggregate performance data from the sub-populations back to the original target audiences. To ensure statistical rigor, the authors implemented a stopping rule based on "normalized regret" (pPVR). The test concludes only when there is 95% confidence that the regret of choosing the current best ad is below a 1% threshold.
Experimental Evaluation
The algorithm was evaluated through both controlled simulations and a real-world deployment on JD.com.
Simulation Results
In a setup with 1,000 replications involving 2 creatives and 2 overlapping TAs, the Thompson Sampler (TS) was compared against standard "Equal Allocation" (A/B/n testing) and industry-standard "Split-Testing." The TS approach significantly minimized total regret and demonstrated superior accuracy:
- TS Accuracy: 85.8% (correctly identified the best ad-audience match at the time of stopping).
- Equal Allocation Accuracy: 77.8%.
- Standard Split-Testing Accuracy: 70.8%.
Real-World Case Study: JD.com
The system was deployed for a major cellphone manufacturer on JD.com’s advertising platform. Over a 6-hour window involving approximately 18,500 users, the algorithm tested 3 creatives across 2 TAs. The results were compelling:
- Increased Performance: The algorithm generated 8.2% more clicks compared to a simulated non-adaptive equal allocation strategy.
- Speed to Confidence: It reached a 98.4% confidence level in the winning creative in just over six hours, demonstrating the efficiency of adaptive learning in a high-traffic environment.
Key Insights
One of the most significant findings is the benefit of Cross-Audience Learning. Because a user in the "Males in SF" group provides data that informs the evaluation of both the "Male" and "SF" target audiences, the algorithm learns more efficiently than if it treated these groups as independent silos. Interestingly, the authors found that as audience overlap increases, the algorithm can actually identify the best ad faster because the cross-audience learning effect becomes more pronounced.
Limitations and Future Directions
Despite its success, the framework faces a "combinatorial explosion" as the number of target audiences increases; $K$ audiences result in $2^K - 1$ sub-populations. Currently, the implementation is optimized for up to 5 TAs and 5 creatives. Additionally, the model assumes independence between user visits and lacks binding budget constraints. Future work may explore reinforcement learning frameworks to incorporate long-term user dependencies and complex budget management.
Conclusion
By reframing the audience evaluation problem as a contextual bandit task, this research provides a robust solution to the overlap problem in digital advertising. The implementation on JD.com proves that adaptive experimentation not only provides more accurate results than traditional A/B testing but also does so more cost-effectively by reducing the regret associated with sub-optimal ad displays.
Summary updated