Heteroscedastic Gaussian Process Q-learning for Sample Efficient Experimentation

Maximilian Bloor, Max Mowbray, Calvin Tsay, Ehecatl Antonio del Rio ChanonaarXiv 2026 · arXiv

TL;DR

HGPQ is a new reinforcement learning (RL) algorithm designed for "self-driving laboratories" where experiments are expensive and data is scarce. By modeling the Q-function with a heteroscedastic Gaussian Process, the algorithm automatically determines the optimal learning rate for every data point, allowing it to master complex chemical and biological control tasks in hundreds of episodes rather than the thousands required by deep RL.

What Problem Does This Paper Address?

The rise of self-driving laboratories (SDLs) promises to accelerate discovery in chemistry and biology by using AI to automate experimental campaigns. However, these systems face a massive hurdle: data efficiency. A single experiment (an "episode") might take hours or days and cost significant resources.

Standard Deep RL algorithms (like SAC or TD3) are notoriously data-hungry, often requiring thousands of trials to converge. While Gaussian Processes (GPs) are the gold standard for small-data regimes, existing GP-based RL methods often struggle with:

  1. Compounding Model Errors: Learning a dynamics model and planning through it can lead to "hallucinations" over long horizons.
  2. Manual Tuning: Standard Q-learning requires a carefully tuned learning rate.
  3. Non-Uniform Noise: In real physical systems, some operating regions are naturally noisier than others. Treating all data with equal "trust" leads to slow learning or instability.

Methods

The authors introduce Heteroscedastic Gaussian Process Q-learning (HGPQ). Instead of learning a model of the physics/chemistry, HGPQ learns the Q-function (the expected long-term reward) directly as a Gaussian Process.

Key Innovations:

  • The Kalman Gain as a Learning Rate: The authors prove that if you treat each Bellman target as a noisy measurement, the optimal learning rate—the one that minimizes mean squared error—is the Kalman gain. This effectively turns Q-learning into a form of Bayesian inference.
  • Heteroscedastic Noise Modeling: This is the core "aha" moment. HGPQ uses a second GP to estimate the noise variance ($\lambda_i$) of each target. This allows the model to distinguish between reliable data and "untrusted" data coming from noisy conditions or regions where the value function has sharp gradients (bootstrap error).
  • Backward Refitting: After each episode, the algorithm refits the GPs for each time step in reverse order (from the end of the experiment to the beginning), allowing information to flow backward efficiently.
Figure 1: The Kalman gain moves the mean towards a "trusted" target (red) much more aggressively than an "untrusted" one (blue).

Datasets

The authors evaluated HGPQ on five simulated benchmarks designed to mimic real-world process control challenges:

  1. Non-stationary Linear MDP: A system where reward coefficients flip signs periodically.
  2. Semi-batch Reactor: A nonlinear chemical process with temperature and volume constraints.
  3. Photosynthetic Biomass Production: A bioprocess model for growing cyanobacteria.
  4. Inverted Pendulum: A classic control task (swing-up).
  5. Target Tracking: Controlling a spring-mass system.

These benchmarks are particularly difficult because they involve continuous states/inputs, process noise, and often only provide a reward at the very end of a batch.

Key Insights and Ideas

  • Self-Tuning Updates: Because HGPQ learns the noise level of the data, it effectively tunes its own learning rate. In regions where the data is reliable, it learns rapidly; where the data is noisy, it remains cautious.
  • Handling "Bootstrap Error": Even in deterministic systems, the targets in Q-learning can be "noisy" if the value function has high gradients (e.g., near a constraint boundary). HGPQ's heteroscedasticity naturally detects these regions and prevents them from corrupting the global estimate.
  • Theoretical Guarantees: The paper provides a high-probability bound on the error of the value estimate that holds uniformly over the explored region, providing a solid mathematical foundation for the method.
Figure 2: HGPQ identifies state-dependent noise (A) and high-gradient "bootstrap error" (B), adjusting the target variance accordingly.

Key Results

HGPQ demonstrated superior sample efficiency across the board.

  • Speed of Learning: Pooled over all benchmarks, HGPQ reached performance thresholds significantly faster than deep RL baselines (SAC, TD3, DQN) and other GP/kernel methods.
  • Threshold Success: HGPQ reached 69% of performance thresholds within just 150 episodes, a feat that took most deep RL baselines over 1,000 episodes to approach.
  • Robustness to Noise: In the chemical reactor and bioprocess benchmarks—where noise is highly dependent on the state—HGPQ significantly outperformed "homoscedastic" versions that assumed a constant noise level.
Figure 3: Comparison of sample efficiency. HGPQ (solid red) reaches higher fractions of performance thresholds in far fewer episodes than baselines.

Limitations and Future Work

  • Computational Complexity: Exact GP regression scales as $O(n^3)$, which can become slow as the number of experiments grows. The authors proposed a "Sparse HGPQ" variant to mitigate this, but scaling to thousands of data points remains a challenge for GPs.
  • Constraint Handling: While the benchmarks included penalties for violating state constraints (like reactor temperature), the algorithm does not yet have a mechanism to guarantee safety during exploration. The authors suggest "feasibility backups" as a future direction.
  • Kernel Dependency: Like all GP methods, performance depends on the choice of kernel. While they used standard Matern kernels successfully, highly discontinuous processes might require more specialized kernel engineering.

Summary updated

Abstract

Self-driving laboratories must learn to control a process from tens to hundreds of expensive experiments, too few for deep reinforcement learning (RL). We learn the Q-function directly with a Gaussian process (GP), which avoids learning the dynamics, and show that the GP posterior supplies the learning rate of Q-learning. If each Bellman target is treated as a noisy measurement of the Q-function, the learning rate that minimises the mean squared error is the Kalman gain, and applying it to every observed transition is equivalent to GP regression. Heteroscedastic Gaussian Process Q-learning (HGPQ) sets the noise variance of each target with a heteroscedastic GP, which learns how unreliable the targets are across the states and inputs. Targets from noisy operating conditions, or from regions where the value function has large gradients, then move the estimate less than targets from low-noise regions, so HGPQ learns quickly where the data are reliable without being misled where they are not. For kernel value iteration with per-target variances that are assigned when a target is collected, we prove a high-probability bound on the error of the value estimate, uniform over the explored region. Pooled over five simulated benchmarks, which include a chemical reactor and a bioprocess, HGPQ approaches the best return found by any method in fewer episodes than GP, kernel and deep RL baselines.
BibTeX
@misc{bloor2026heteroscedastic,
  author = {Maximilian Bloor and Max Mowbray and Calvin Tsay and Ehecatl Antonio del Rio Chanona},
  title = {{Heteroscedastic Gaussian Process Q-learning for Sample Efficient Experimentation}},
  year = {2026},
  eprint = {2611.21249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21249},
}
Figure