Uncertainty-Aware Hybrid Offline Reinforcement Learning with Online Action Selection Enabled by Sparse Gaussian-Process Critics

Marco Chan, Christos Evripidou, Liqiu Dong, Benoit Chachuat, Mehmet MercangözarXiv 2026 · arXiv

In the world of industrial process control, "start-ups" and "grade changes" are the high-stakes moments. These transitions are often still handled manually by expert operators because building accurate physical models for these transient states is incredibly difficult. Offline Reinforcement Learning (RL) offers a tempting alternative: learn how to automate these transitions directly from historical data.

However, as a new paper by Marco Chan and colleagues at Imperial College London highlights, the biggest threat to offline RL is distribution shift—the tendency for an AI to confidently suggest a "brilliant" action that it has never actually seen before, often leading to catastrophic failure in the real world.

TL;DR

The authors improve the HOFLON offline RL framework by replacing its complex two-model architecture with a single, uncertainty-aware Sparse Gaussian Process (sGP) critic. By using a "pessimistic" lower confidence bound for action selection, the new HOFLON-sGP method eliminates failed reactor start-ups and achieves higher returns than previous methods while simplifying the system.

What Problem Does This Paper Address?

Offline RL learns from a fixed dataset of historical transitions. The core challenge is that if you simply ask a model to "maximize the predicted reward," it will often pick state-action pairs that are poorly represented in the training data. Why? Because the model's predictions are most likely to be wildly (and erroneously) optimistic in regions where it has no data.

In a chemical plant, this isn't just a math problem; it's a safety hazard. An over-optimistic controller might push a reactor into a "thermal runaway" state because it incorrectly predicts a high reward for an aggressive cooling strategy it hasn't fully understood. The original HOFLON framework tried to fix this by using a secondary model—an Adversarial Autoencoder (AAE)—to penalize "unfamiliar" actions. While effective, this added complexity and required tuning a sensitive penalty weight that didn't have a clear physical unit.

Methods

The researchers introduced HOFLON-sGP, which streamlines the architecture by using a Sparse Variational Gaussian Process (sGP) as the "critic."

Unlike standard neural networks or tree ensembles, a Gaussian Process provides two things for every prediction:

  1. The Mean ($\mu$): The expected return (reward).
  2. The Standard Deviation ($\sigma$): A mathematically grounded measure of uncertainty.

Instead of just maximizing the mean, HOFLON-sGP maximizes a Lower Confidence Bound (LCB): $\mu - \kappa\sigma$. The parameter $\kappa$ acts as a "pessimism" dial. By subtracting the uncertainty, the controller is naturally discouraged from taking actions that aren't well-supported by historical data.

Structure of HOFLON vs HOFLON-sGP

This is a "one-step" method, meaning it evaluates the immediate next action assuming the plant will return to historical operating practices afterward. This avoids the "compounding error" problems often seen in more complex multi-step RL algorithms.

Datasets

The method was evaluated on a polymerization-reactor start-up benchmark. The goal is to bring polymer concentration to a specific set-point while maintaining a stable temperature. The training data consisted of 100 historical start-ups controlled by human-tuned PI (Proportional-Integral) controllers with randomized gains, simulating the varied and sometimes suboptimal data found in real industrial historians.

Key Insights and Ideas

  • The "Tie-Breaker" Mystery: The authors discovered that in the original HOFLON, the AAE penalty was tiny—four orders of magnitude smaller than the critic's values. Yet, without it, the system often failed. They realized that because tree-based models (like XGBoost) produce "flat" piecewise-constant predictions, the AAE acted as a crucial tie-breaker, nudging the optimizer toward better-supported actions when the reward predictions were identical.
  • Pessimism is Mandatory: When the researchers set $\kappa=0$ (ignoring uncertainty), the system failed in 13 out of 15 runs, often ending in thermal runaway. In safety-critical engineering, being "optimistic in the face of uncertainty" is dangerous.
  • Kernel Choice is a Design Decision: The choice of the GP's kernel (the function defining how points relate) significantly impacted stability. The Matérn-5/2 kernel proved more robust than the standard Squared-Exponential kernel, requiring less aggressive pessimism to keep the reactor within safe operating bounds.

Key Results

  • Zero Failures: With the Matérn-5/2 critic and a pessimism level of $\kappa=3$, HOFLON-sGP achieved zero failures across all test runs.
  • Superior Returns: It achieved a median return of 13,464, outperforming the original HOFLON (13,393) and significantly beating the best historical runs in the dataset (12,327).
  • Better Data Support: The sGP version stayed much closer to the "known" operating regions. During steady-state operation, it had 0% off-support steps, whereas the original HOFLON often drifted into unfamiliar territory.
  • Faster Computation: By consolidating into a single model, the online optimization became faster, making it more practical for real-time deployment.
Closed-loop performance comparison

Limitations and Future Work

The authors acknowledge that this study was conducted on a single simulator with fixed initial conditions. Real-world plants face non-stationary disturbances and varying starting states that weren't fully explored here.

Furthermore, because this is a "one-step" improvement method, it is inherently conservative. It can't easily discover radical new multi-step strategies that are completely different from historical practice. Future work could involve "bootstrapping"—using the GP's uncertainty to safely allow the model to learn and improve its policy over multiple iterations without leaving the "safety envelope" of the data.

Summary updated

Abstract

Start-ups and grade changes remain largely operator-driven, while developing and maintaining dynamic models for model-based control requires substantial engineering effort. Hybrid Offline Learning and Online Optimisation (HOFLON) uses one-step offline reinforcement learning to select actions from historical transition data. In its original implementation, the online objective combines the predicted return from a tree-ensemble critic with an adversarial-autoencoder (AAE) penalty for unfamiliar state-action pairs. We examine the effect of this penalty and replace the critic and AAE with a sparse variational Gaussian-process (sGP) critic. Actions maximise $μ- κσ$, an uncertainty-adjusted estimate of the return from one action followed by the historical operating practice. On the HOFLON polymerisation-reactor benchmark, removing the AAE from HOFLON led to 4 failed start-ups in 15 runs over five critics. At its published weight, the variation in the AAE penalty is four orders of magnitude smaller than the variation in the tree-ensemble critic. Nevertheless, it changes actions when the predicted returns of competing candidates are nearly equal, and paired runs with and without the penalty diverged from the first move. With a Matérn-5/2 sGP critic, $κ= 0$ led to failure in 13 of 15 runs, whereas with $κ= 3$ none of 15 runs (five critics, three noise realisations) failed. The median and lowest returns were 13,464 and 13,398 (HOFLON: 13,393 and 12,411). A squared-exponential sGP critic with learned inducing inputs needed $κ\geq 3$ to remain within the empirical data support in steady state, and then settled with a temperature offset of 2.0 K.
BibTeX
@misc{chan2026uncertainty,
  author = {Marco Chan and Christos Evripidou and Liqiu Dong and Benoit Chachuat and Mehmet Mercangöz},
  title = {{Uncertainty-Aware Hybrid Offline Reinforcement Learning with Online Action Selection Enabled by Sparse Gaussian-Process Critics}},
  year = {2026},
  eprint = {2611.14374},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14374},
}
Figure