contextual: Evaluating Contextual Multi-Armed Bandit Problems in R

Robin van Emden, Maurits KapteinarXiv 2018 · arXiv

Abstract

The contextual package for R provides a powerful, object-oriented framework for simulating and evaluating Contextual Multi-Armed Bandit (CMAB) algorithms. By bridging the gap between theoretical research and empirical testing, it offers a standardized environment for everything from simple synthetic simulations to massive, parallelized offline evaluations on real-world datasets. This paper details the architecture of the package, its implementation of offline evaluation methods, and its performance in replicating landmark studies.

Introduction

In sequential decision problems—such as clinical trials, online advertising, and personalized medicine—agents must choose between multiple options (arms) to maximize a cumulative reward. This necessitates balancing the explore-exploit dilemma: deciding whether to stick with the best-known option (exploit) or try a new one to gather more information (explore). While traditional bandits treat every decision identically, Contextual Multi-Armed Bandits (CMAB) utilize side information, or context (e.g., user demographics), to inform choices.

Despite the growing importance of CMAB algorithms, researchers have lacked a standardized, extensible tool in the R environment to compare them fairly. Existing solutions were often limited or opaque. The contextual package addresses this by providing a transparent, high-performance playground for researchers and practitioners to develop and benchmark bandit policies.

Methods

The contextual package is built using the R6 object-oriented system, ensuring a modular and extensible codebase. The architecture is centered around six core classes that define the simulation environment and the decision-making process.

The architectural framework of the contextual package, showing the workflow between the Agent, Policy, and Bandit components.

The primary components include:

  • Bandit: Responsible for generating the environment, context, and rewards, either through synthetic distributions or from historical datasets.
  • Policy: The algorithmic "brain" (e.g., LinUCB, Thompson Sampling) that selects an arm based on the provided context.
  • Agent: Acts as the bridge, managing the interaction loop between a specific Bandit and a Policy.
  • Simulator: The execution engine that runs agents, manages parallelization via the doParallel package, and ensures reproducibility through seed management.
  • History & Plot: Tools for logging interactions and visualizing performance metrics such as cumulative regret and reward rates.

A significant feature of the package is its implementation of the "Replay Method" for offline evaluation. This technique allows researchers to test new policies on historical data by filtering for events where the new policy's choice matches the choice recorded by the original logging policy, providing a counterfactual assessment without live deployment.

Datasets

The paper demonstrates the package using two primary data modalities:

  1. Synthetic Bandits: Built-in generators for Bernoulli, Gaussian, and Poisson distributions, ideal for controlled experiments and sanity checks.
  2. Real-World Benchmarks: The package was stress-tested using the Yahoo! Today Module Dataset, which contains approximately 45.8 million interaction events. A subset of 37.4 million events was used to evaluate scalability and replicate historical findings.

Results and Key Insights

The contextual package demonstrates that R can effectively handle "big data" bandit simulations. By integrating with high-performance databases like MonetDB and utilizing parallel processing, the authors processed tens of millions of events in under 24 hours. Specifically, a simulation involving 42 agents over 37.4 million events was completed in 22 hours on a 64-core server.

The package successfully replicated the core findings of the landmark study by Li et al. (2010), confirming the relative performance of various Upper Confidence Bound (UCB) and Epsilon-Greedy policies. Interestingly, long-term evaluation (8 days vs. 1 day) revealed that some performance leads, such as Hybrid LinUCB over Disjoint LinUCB, could reverse over time, highlighting the necessity of extended evaluation horizons.

Visual analysis provided by the package's plotting tools also revealed "sinusoidal" reward patterns in the Yahoo! data. These patterns reflect the natural daily cycles of web traffic, a nuance that summarized statistics might otherwise obscure.

Conclusion and Future Work

The contextual package provides a much-needed standardized interface for CMAB research in R, ensuring that algorithms are compared under identical conditions. While the current "Replay" method is a robust standard, future iterations of the package aim to include more data-efficient "doubly robust" estimators. Additionally, the authors plan to expand support to dueling bandits and combinatorial bandits, and to establish a community library for sharing new policy implementations. This evolution will transform contextual into a comprehensive, living library for the bandit research community.

Summary updated

Abstract

Over the past decade, contextual bandit algorithms have been gaining in popularity due to their effectiveness and flexibility in solving sequential decision problems---from online advertising and finance to clinical trial design and personalized medicine. At the same time, there are, as of yet, surprisingly few options that enable researchers and practitioners to simulate and compare the wealth of new and existing bandit algorithms in a standardized way. To help close this gap between analytical research and empirical evaluation the current paper introduces the object-oriented R package "contextual": a user-friendly and, through its object-oriented structure, easily extensible framework that facilitates parallelized comparison of contextual and context-free bandit policies through both simulation and offline analysis.
BibTeX
@misc{emden2018contextual,
  author = {Robin van Emden and Maurits Kaptein},
  title = {{contextual: Evaluating Contextual Multi-Armed Bandit Problems in R}},
  year = {2018},
  eprint = {1811.01926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.01926},
}
Figure