Abstract
Evaluating interactive machine learning systems through Off-Policy Evaluation (OPE) is critical for minimizing the risks and costs associated with live A/B testing. However, the field has been hindered by a lack of large-scale, public datasets with ground-truth labels. This paper introduces the Open Bandit Dataset and Pipeline, a massive collection of over 26 million records from ZOZOTOWN, Japan's largest fashion e-commerce platform. By providing logs from multiple behavior policies alongside their ground-truth performance, this dataset enables fair and reproducible benchmarking of OPE estimators and bandit algorithms.
Introduction
Evaluating recommendation systems is inherently challenging because we only observe user reactions to the specific items presented. While A/B testing is the gold standard, it is often impractical due to costs and potential negative impacts on user experience. Off-Policy Evaluation (OPE) offers a solution by estimating the performance of new policies using historical log data. Despite theoretical progress, OPE research has lacked standardized benchmarks, often relying on synthetic data or proprietary datasets. The Open Bandit Dataset addresses this gap by providing a real-world, large-scale benchmark for the community.
The Open Bandit Pipeline and OPE Methods
The authors provide the Open Bandit Pipeline, an open-source suite designed for dataset preprocessing and policy simulation. The pipeline includes implementations of three foundational OPE estimators:
- Direct Method (DM): Uses a supervised model to predict rewards for context-action pairs. It is effective but can be biased if the model is misspecified.
- Inverse Probability Weighting (IPW): Reweights rewards based on the probability ratio between the new and old policies. It is unbiased but can suffer from high variance.
- Doubly Robust (DR): A hybrid approach that combines DM and IPW to balance bias and variance, often providing the most robust results.
The Open Bandit Dataset
The dataset was collected during a seven-day experiment on ZOZOTOWN in November 2019. It is structured into three distinct campaigns: "All items," "Men's," and "Women's." A unique feature of this dataset is the inclusion of logs from two different behavior policies: a Random policy and a Bernoulli Thompson Sampling (Bernoulli TS) policy. This dual-policy structure allows researchers to use one policy's logs to predict the other's performance, comparing the result against the known ground truth.
Key Results and Benchmarking
The dataset establishes clear ground-truth Click-Through Rates (CTR) for the behavior policies. As shown in the data summary, the Bernoulli TS algorithm consistently outperformed the Random baseline across all campaigns. For example, in the "ALL" campaign, the Random policy yielded a 0.35% CTR, while the Bernoulli TS policy achieved 0.50%. These figures serve as the target for OPE algorithms, allowing for objective measurement of estimator accuracy in a real-world setting. Future researchers can take the logs generated by the Random policy, apply their OPE estimators to simulate the Bernoulli TS policy, and see how close their predicted CTR gets to the actual ground-truth.
Discussion and Future Work
The release of the Open Bandit Dataset represents a significant step toward an "ImageNet moment" for bandit algorithms, providing the scale and complexity needed to move beyond toy problems. While the current data focuses on context-free policies, it captures the inherent noise and sparsity of real-world user behavior. Future iterations of the dataset will include log data from personalized contextual bandit policies, further expanding the utility of the benchmark for modern recommendation systems.
Summary updated