Introduction
In many real-world scenarios, systems must be optimized across various "contexts" or "tasks." For example, a database requires tuning for different workloads, or a nuclear reactor needs distinct control settings depending on the current state of its plasma.
Typically, "nature" dictates the context, and the system merely reacts—a setup known as Online Contextual Bandits. However, in many engineering fields, researchers have access to high-fidelity simulators. This creates an "offline" opportunity: before deploying a controller in the real world, we can use the simulator to "practice" on whichever tasks we choose. The primary challenge is that these simulations are often incredibly expensive, sometimes taking hours for a single run.
The core question this paper addresses is: Given a limited "budget" of simulations, which tasks should you practice on, and which actions should you test to build the best overall control policy? To solve this, the authors introduce Multi-Task Thompson Sampling (MTS), a Bayesian Optimization framework designed specifically for these offline scenarios.
Multi-Task Thompson Sampling (MTS)
While standard Bayesian Optimization (BO) usually searches for a single global maximum, MTS aims to find the optimal action for every task within a given set.
- The Algorithm: At each step, the MTS algorithm draws a sample function from a Gaussian Process (GP) posterior. It then identifies the task where the "gap" between the current best-known reward and the sampled potential reward is the largest. It performs the simulation for that specific task and action, and subsequently updates its model.
- Theoretical Guarantees: The authors provide a regret bound proving that the algorithm’s performance converges to the optimum at a rate similar to single-task Thompson Sampling. This ensures that no single task is left with exceptionally poor results.
- Non-Stationary Modeling: A major contribution of this work is the use of the Gibbs kernel. Standard kernels (like RBF) assume the "smoothness" of the reward function is uniform everywhere. However, in multi-task settings, some tasks are "easy" (flat landscapes) while others are "hard" (peaky landscapes). The Gibbs kernel allows the model's "lengthscale" to vary by task, preventing the algorithm from wasting valuable simulation resources over-exploring simple tasks.
Experimental Validation & Results
The paper evaluates the MTS method using both synthetic benchmarks (modified versions of classic optimization functions like Branin-Hoo and Hartmann across various dimensions) and a high-fidelity "predictive mode" simulator for tokamaks (TRANSP). The goal in the nuclear fusion simulator was to optimize neutral beam controls across 7 different plasma states to maximize stability and fusion efficiency.
Key Insights
- Active Task Selection: The authors demonstrate that if a model poorly captures task difficulty, actively picking tasks can actually perform worse than random selection. This highlights that intelligent algorithms require intelligent underlying models, such as the Gibbs kernel.
- Resource Allocation: MTS naturally learns to stop querying "solved" tasks. Once the reward for a specific plasma state or synthetic function levels off, the algorithm shifts its focus to other states where improvement is still predicted.
- Joint Learning Matters: By using a single GP to model all tasks simultaneously (Joint-MTS), the algorithm successfully "transfers" knowledge between similar tasks, outperforming versions that treat every task as a completely isolated problem.
Performance
In synthetic benchmarks, MTS consistently outperformed baselines like Multi-task Expected Improvement (MEI), standard Thompson Sampling (TS), and Knowledge Gradient (REVI), particularly in higher-dimensional spaces.
In the TRANSP nuclear fusion simulator, the Joint-MTS approach reached higher rewards (better stability and efficiency) faster than standard Thompson Sampling. The algorithm successfully identified "difficult" plasma states that required more exploration while avoiding redundant simulations in states that were already well-optimized.
Limitations and Future Directions
While highly effective, the authors acknowledge a few limitations:
- Theoretical Slack: The theoretical regret bound has an extra dependence on the number of tasks, which might be "loose" and could potentially be tightened in future mathematical proofs.
- Myopic Nature: Like most BO methods, MTS is "myopic"—it selects the best task for the next immediate step rather than planning a long-term sequence of experiments.
- Scaling to Continuous States: The primary focus and theoretical results of this paper are for finite task sets. Expanding this framework to fully continuous, high-dimensional state spaces for Reinforcement Learning (RL) is a promising next step for future research.
Summary updated