Introduction
Contextual Multi-Armed Bandits (MABs) serve as a foundational framework for sequential decision-making under uncertainty, powering modern recommendation engines and personalized systems. A persistent challenge in this domain is the "cold start" problem: new models must explore extensively to estimate reward parameters, incurring significant "regret" (lost potential reward) during the initial learning phase.
While transfer learning offers a potential solution by leveraging knowledge from related tasks, traditional methods often require access to the raw data or context logs of previous tasks. In many real-world scenarios—such as healthcare or finance—this is prohibited by privacy regulations or storage constraints. This paper addresses the challenge of Hypothesis Transfer Learning (HTL) in bandits, where a learner must improve performance on a target task using only the parameters of pre-trained source models, without access to their underlying data.
Methodology: Weighted LinUCB
The authors introduce Weighted LinUCB, an evolution of the classic Linear Upper Confidence Bound algorithm. The core innovation lies in estimating the optimal bandit parameters ($\theta$) as a dynamic convex combination of a pre-trained source model ($\theta_S$) and a target model ($\hat{ heta}_T$) being learned in real-time:
$$\hat{ heta} = \alpha_S \theta_S + \alpha_T \hat{ heta}_T$$
To manage the weights ($\alpha$), the authors propose two distinct strategies:
- Hard Update Strategy: At each decision interval, the algorithm evaluates the confidence sets of both the source and target models. It selects the model with the lower uncertainty (tighter confidence bound) to guide action selection. This effectively switches from the source to the target once the target model becomes sufficiently informed.
- Softmax Update Strategy: This approach utilizes a "softer" convex combination of all available models. The weights are updated using a Softmax function regularized by KL-divergence, which prevents erratic weight fluctuations and allows for a smoother transition between hypotheses.
Crucially, these strategies are designed to be immune to negative transfer. If a source model is irrelevant or poorly matched to the target task, the algorithm rapidly diminishes its weight, ensuring that performance never degrades below the baseline of a standard LinUCB algorithm.
Theoretical and Empirical Evaluation
Theoretical Insights
The researchers mathematically established that the "Hard Update" strategy maintains a regret bound at least as good as the standard LinUCB. They identified a critical time threshold, $\kappa$: before this point, the source model typically provides more reliable guidance; after $\kappa$, the target model—built on task-specific data—takes precedence. By leveraging the source model during the $t < \kappa$ phase, the algorithm significantly reduces the exploration required, thereby lowering cumulative regret.
Experimental Setup
The framework was validated using two distinct environments:
- Synthetic Data: Using context vectors drawn from a multivariate Gaussian distribution ($d=20$) and 1,000 arms, the authors controlled the similarity between source and target tasks to test the algorithm's robustness against varying degrees of relatedness.
- MovieLens Dataset: To simulate a real-world recommendation scenario, the authors used movie genre vectors (18-dimensional) and user demographic traits. They treated a model trained on one user group as a "source" for a new, similar user group to evaluate the practical utility of parameter-based transfer.
Key Results
The experiments demonstrated that Weighted LinUCB consistently outperforms both the standard LinUCB and the EXP4 algorithm. The performance gains were most pronounced during the early "cold start" phase, where the transferred hypothesis provided a strong starting point. Furthermore, the algorithm successfully avoided negative transfer when presented with "bad" source models, recovering the standard learning rate once the lack of fit was detected.
Conclusion and Future Work
This work demonstrates that hypothesis transfer is a viable and privacy-preserving method for accelerating learning in contextual bandits. By sharing only model parameters, organizations can collaborate or reuse knowledge without compromising data security.
While the current framework assumes a linear relationship between contexts and rewards, future research directions include adapting these weighted models to kernelized bandits to handle non-linear reward structures. Additionally, the authors suggest exploring non-stationary environments where source models may need to be dynamically updated or discarded as the underlying data distribution shifts over time.
Summary updated