In the world of automated decision-making, contextual bandits are the gold standard for balancing exploration (trying new things) and exploitation (sticking with what works). However, these models typically assume that the environment they learn in is the same one they’ll be deployed in. In reality, collecting data in the "target" environment—like testing a new drug on humans—is often prohibitively expensive or risky.
A new paper, "Towards Domain Adaptive Neural Contextual Bandits," introduces DABand, the first general framework designed to train a bandit model on a low-cost "source" domain (like lab mice) and successfully adapt it to a high-cost "target" domain (like humans) without ever seeing a single reward in the target setting.
TL;DR
DABand is the first general method for domain adaptation in contextual bandits, allowing models to learn optimal decision policies from cheap source data and apply them to expensive target domains. By combining adversarial representation alignment with novel reward regularization, the authors prove that their algorithm maintains a sub-linear regret bound even across shifting distributions.
What Problem Does This Paper Address?
Contextual Bandit (CB) algorithms usually fail when faced with distribution shift. If you train a recommendation engine on data from one country and deploy it in another, the "context" (user behavior) and "rewards" (click-through rates) change.
The authors identify a specific, high-stakes challenge: Cross-domain exploration. How do you explore a high-cost target domain by only collecting feedback from a low-cost source domain? This presents a "chicken-and-egg" dilemma: to explore effectively, you need good feature representations, but to learn good representations, you need to explore. Furthermore, aligning these domains is difficult because, in a bandit setting, you don't know the "ground truth" reward for actions you didn't take.
Methods
DABand builds on Neural-LinUCB, which uses a neural network to extract features and a linear model to make bandit decisions. To bridge the domain gap, the authors introduce a multi-part objective function:
- Source Regret: Standard bandit learning to minimize loss on the source data.
- Representation Alignment (Data Divergence): Using an adversarial discriminator (similar to a GAN), the model forces the neural encoder to produce latent representations that look identical for both source and target contexts.
- Regression Error: An $L_1$ loss term that ensures the model accurately predicts rewards in the source domain, providing a stable foundation for transfer.
- Predicted Reward Regularization: A crucial novel term that penalizes the model for overestimating rewards in the target domain. Since the model never sees target rewards, it can become "over-optimistic"; this term keeps the model's expectations in check.
The training follows a minimax optimization: the encoder tries to fool the discriminator while the bandit parameters try to minimize regret.
Datasets
The authors evaluated DABand on three increasingly complex image-based bandit benchmarks:
- DIGIT: Adapting from MNIST (grayscale digits) to MNIST-M (colored digits on noisy backgrounds).
- VisDA17: A massive domain adaptation dataset moving from synthetic 3D renders (source) to real-world photos (target) across 12 object classes.
- S2RDA49: A "Synthetic-to-Real" dataset featuring 49 classes, where the authors focused on a 10-class subset to match the VisDA17 labels.
Key Insights and Ideas
- The $H\Delta H$ Distance for Regression: Most domain adaptation theory focuses on classification. The authors successfully generalized the $H\Delta H$ divergence—a measure of how different two distributions are—to the regression/bandit setting.
- Why Naive Alignment Fails: The paper reveals that simply aligning features (the standard "Domain Adaptation" approach) isn't enough for bandits. Without the "Regression Error" and "Predicted Reward" terms, the model often overfits the source domain or becomes wildly inaccurate in the target domain.
- Two-Way Feedback: Interestingly, the model creates a feedback loop. Source rewards help the encoder find relevant features for the target, while target contexts help the source model explore "arms" (actions) that are more likely to be relevant in the final deployment environment.
Key Results
DABand significantly outperformed state-of-the-art baselines like LinUCB and Neural-LinUCB across all tests.
- Accuracy Boost: On the VisDA17 dataset, DABand achieved 46.4% average accuracy in the zero-shot target setting, while the standard Neural-LinUCB struggled at just 10.0%.
- Regret Reduction: In "continued training" scenarios (where the model eventually starts seeing target rewards), DABand provided a much better "head start," resulting in significantly lower cumulative regret over time compared to starting from scratch.
- Ablation Success: Removing the novel "Predicted Reward" regularization caused performance to drop significantly (e.g., from 46.4% to 43.0% on VisDA17), proving its necessity.
Limitations and Future Work
- The Interpretability Trade-off: While DABand performs well, the use of deep neural encoders makes the model less interpretable than classic linear bandits.
- Alignment Dependency: The method's success relies heavily on the quality of the alignment. If the source and target domains are too different to be mapped into a shared latent space, the transfer will fail.
- Future Directions: The authors suggest exploring Bayesian deep learning to improve uncertainty quantification and investigating more sophisticated ways to handle high-dimensional, dense target contexts.
Check out the code: https://github.com/Wang-ML-Lab/DABand
Summary updated