Abstract
Evaluating ad creatives across overlapping target audiences presents a significant challenge for digital advertisers. Traditional A/B testing methods struggle with statistical bias and inefficiency when users belong to multiple target groups. This paper proposes a novel framework that transforms overlapping audiences into disjoint sub-groups, allowing for a contextual multi-armed bandit approach using Thompson Sampling. This method enables "cross-audience learning," where data from a single sub-group informs multiple target audiences simultaneously. Simulations and real-world deployment on JD.com demonstrate that this approach identifies optimal ad-audience matches faster and with lower regret than standard testing methods.
Introduction: The Overlap Problem
In digital advertising, target audiences (TAs) are rarely mutually exclusive. An advertiser might target "San Francisco users" and "Male users," creating a significant overlap of "San Francisco Males." Traditional split-testing (A/B/n testing) fails to handle this overlap efficiently. Randomly assigning overlapping users to a single group loses valuable cross-contextual data, while non-adaptive testing continues to waste budget on underperforming ads until the experiment concludes.
Methodology: From Overlap to Disjoint Contexts
The authors introduce a two-step framework to resolve these issues. First, overlapping TAs are partitioned into a set of Disjoint Audiences (DAs). For example, two overlapping TAs are split into three distinct DAs: those in TA1 only, those in TA2 only, and those in both.
Second, the system employs Contextual Thompson Sampling (TS). Each DA serves as a "context," and each ad creative is an "arm." The algorithm maintains a Beta distribution for each arm's success rate, choosing ads based on their probability of being optimal. Crucially, the algorithm uses probabilistic aggregation to map learned performance from DAs back to the original TAs. This enables faster learning through shared data points; a single click from a user in the overlap informs the performance metrics of both parent TAs.
Experimental Results
The framework was tested through 1,000 simulated replications comparing TS against Equal Allocation (EA) and standard Split-Testing (ST). The results consistently favored the adaptive bandit approach across multiple dimensions of efficiency and accuracy.
Key results from the simulation include:
- Higher Accuracy: TS correctly identified the best creative-audience match 85.8% of the time, significantly outperforming EA (77.8%) and ST (70.8%).
- Lower Regret: As shown in Figure 2(a) and (b), TS minimizes "regret"—the opportunity cost of showing suboptimal ads—much more effectively than static methods as more data is collected.
- Efficiency at Stopping: The test stops when the potential loss from not picking the best ad falls below a 1% threshold. TS reached this threshold with smaller sample sizes and lower total regret compared to competitors.
Real-World Deployment and Impact
The system is currently deployed on JD.com, a major Chinese e-commerce platform. In a case study involving a cellphone manufacturer's campaign with 2 TAs and 3 creatives, the TS algorithm generated 8.2% more clicks (52 additional clicks over a 6-hour window) than an Equal Allocation strategy would have, demonstrating immediate economic value.
Limitations and Future Work
Despite its success, the approach faces a "combinatorial explosion" as the number of TAs increases, leading to an exponential growth in the number of DAs. Current implementations limit the number of TAs and creatives to five each to maintain computational feasibility. Additionally, the model currently assumes static user arrival rates and auction dynamics. Future work aims to address ad fatigue and dynamic environments using more general reinforcement learning frameworks.
Conclusion
By re-framing overlapping audience testing as a contextual bandit problem on disjoint sub-populations, this research provides a statistically sound and economically efficient alternative to traditional A/B testing. The implementation of cross-audience learning allows advertisers to optimize their campaigns faster while minimizing the costs associated with experimentation.
Summary updated