Causal Contextual Bandits with Targeted Interventions

Chandrasekar Subramanian, Balaraman RavindranICLR 2022

Introduction

In a standard contextual bandit setting, an agent observes a user (the context), picks an action (an arm), and receives a reward. This framework is widely used in recommendation systems and medical trials. However, current models suffer from two major limitations. First, they rely on passive observation, assuming the environment dictates the context. In many real-world scenarios, such as software A/B testing, researchers can actively target specific groups (e.g., "only show this feature to iOS users") to learn faster. Second, most models ignore causal structure, treating different contexts as independent problems. In reality, variables are often causally linked. Knowing that an email subject affects the open rate provides a structural hint that can "leak" valuable information from one experiment to another.

This paper introduces a novel framework for Causal Contextual Bandits with Targeted Interventions. It allows agents to use causal side-information and actively choose which sub-populations to experiment on during training, answering a critical question: How can an agent best allocate a limited budget of targeted experiments to learn a policy that performs optimally across the entire population?

Methodology: The Unc_CCB Algorithm

The authors propose modeling the environment as a Causal DAG (Directed Acyclic Graph). During a training phase of $T$ rounds, the agent can choose both the action ($X$) and the specific context subset ($C^{tar}$) to intervene upon. The model also accounts for auxiliary variables ($C^{other}$) that are visible during training but unavailable during final deployment.

To navigate this environment, the authors introduce the Unc_CCB Algorithm, which operates in two phases:

  1. Phase 1 (Exploration): The agent performs uniform random exploration to establish baseline beliefs.
  2. Phase 2 (Targeted Learning): The agent uses a novel Unc (Uncertainty) measure to pick the most informative (action, context) pairs.

Unlike standard entropy, which treats all uncertainty equally, the Unc measure weights uncertainty by the potential reward of a context. This value-aware exploration ensures the agent doesn't waste its budget exploring contexts unlikely to yield high rewards. By focusing on high-value sub-populations, the agent reduces regret significantly faster. Furthermore, because the agent understands the causal graph, an experiment on one specific context helps update beliefs about related contexts—turning "information leakage" into a powerful feature for sample efficiency.

Experimental Setup and Key Results

The paper evaluates the algorithm using two primary approaches: purely synthetic data (a 4-node causal graph designed to test information leakage) and a CRM sales data-inspired experiment (calibrated with proprietary sales data from Freshworks Inc.).

Superior Efficiency at Low Budgets

In synthetic tests, Unc_CCB consistently outperformed five different baselines, including Thompson Sampling and uniform exploration variants. The performance gap was most pronounced when the training budget ($T$) was small—exactly the scenario where real-world experiments are most expensive.

Regret vs. Number of training rounds

Real-World Impact

In the CRM sales simulation, the algorithm demonstrated substantial practical value. Unc_CCB achieved a mean regret approximately 35% lower than the next best baseline at a training budget of $T=24$.

Normalized Regret vs. Number of training rounds

Beyond empirical success, the authors provided theoretical guardrails, proving a regret bound that shows as the training budget increases, the regret of the learned policy approaches zero at a rate of $1/\sqrt{T}$.

Limitations and Future Work

While Unc_CCB bridges the gap between reactive bandits and active learning, it has a few limitations:

  • Known Graph Assumption: The current algorithm assumes the causal DAG is known beforehand. In many practical scenarios, the graph might be unknown or only partially understood.
  • Categorical Variables: The formalism currently focuses on categorical context variables. Extending this framework to continuous or high-dimensional feature spaces remains an open challenge.
  • Distributional Shifts: The paper assumes the underlying causal model stays the same between training and evaluation. Future work could explore how to make these agents robust to environment shifts where the relationships between variables change over time.

Summary updated

BibTeX
@inproceedings{subramanian2022causal,
  author = {Chandrasekar Subramanian and Balaraman Ravindran},
  title = {{Causal Contextual Bandits with Targeted Interventions}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
}
Figure