A trust layer for fast pedestal-stability surrogates: calibrated error bars, out-of-distribution alarms and few-shot recalibration

Amin VakhshouriarXiv 2026 · arXiv

TL;DR

This paper introduces a "trust layer" for machine learning surrogates used in nuclear fusion, providing calibrated error bars, out-of-distribution (OOD) alarms, and a path for few-shot recalibration. By wrapping fast surrogates in this layer, researchers can finally use millisecond-level stability predictions for high-stakes engineering decisions, knowing exactly when to trust the model and when to revert to expensive physics codes.

What Problem Does This Paper Address?

In nuclear fusion research, predicting the stability of the "pedestal" (the edge region of the plasma) is crucial for reactor design. High-fidelity physics codes like MISHKA or BOUT++ are accurate but painfully slow, taking minutes to hours per simulation. While machine learning (ML) surrogates can provide answers in milliseconds, they are typically "black boxes" that offer point estimates without reliable uncertainty.

The stakes are high: an engineer designing a new tokamak (like SPARC or ITER) is working with plasma shapes that are, by definition, outside the training data of existing surrogates. Standard ML models fail silently in these scenarios. This paper addresses the "missing piece" for deployment: a model-agnostic framework that guarantees reliability even under the "covariate shift" seen when moving between different reactor designs.

Methods

The authors propose a trust layer that wraps any existing surrogate model. It consists of three main components:

  • Conformal Prediction: Specifically "split conformal intervals" and "Mondrian (per-range) calibration." Unlike Bayesian methods, these provide distribution-free coverage guarantees—meaning the error bars are mathematically guaranteed to contain the true value a certain percentage of the time (e.g., 90%) under the assumption of exchangeability.
  • OOD Alarms: Two detectors—Deep Ensemble Spread (measuring model disagreement) and Mahalanobis Distance (measuring how "far" an input is from the training data)—flag cases that are too unfamiliar for the surrogate to handle.
  • Few-shot Recalibration: A method to "tune" the surrogate's error bars for a new device using only a handful (approx. 30) of expensive physics simulations.

A standout feature of this work is the use of pre-registered hypotheses. The authors committed to their evaluation plan before seeing the results, a rigorous practice borrowed from clinical trials to prevent "cherry-picking" successful ML outcomes.

Datasets

  • KARHU: A large public dataset of 38,879 JET-like equilibria with ideal-MHD growth rates computed via the MISHKA code.
  • BOUT++: A resistive MHD dataset based on the CBM18 equilibrium, used for case studies on real-machine equilibria.
  • Real-Machine Equilibria: Data from SPARC, DIII-D, NSTX, and ITER were used to test cross-machine generalization and OOD detection.

Key Insights and Ideas

  • Surrogates Track Their Own Error: The trust layer doesn't make a mediocre model "better" in terms of accuracy, but it makes it honest. A more accurate surrogate will have narrower error bars, while a less accurate one will have wider ones, but both will maintain the target coverage (e.g., 90%).
  • The "Knee" at 30 Labels: For a new reactor design, the authors found a clear point of diminishing returns. Recalibrating with just 30 new simulations significantly restores the reliability of the error bars; adding more (up to 100) provides only marginal improvements.
  • Trust Layer as a Debugger: During development, the trust layer's diagnostics exposed architectural flaws in the authors' own neural operator, such as a radial grid that was too coarse at the pedestal edge. Fixing these "input defects" closed the performance gap between complex and simple architectures.
  • Routing as a Safety Valve: Instead of forcing the surrogate to guess on OOD data, the system can route "scary" cases back to the slow physics code. This ensures that the surrogate only handles what it knows, maintaining high overall system reliability.
Figure 1: Coverage under plasma shape shifts and recalibration.

Key Results

  • Reliable Coverage: In-distribution, the conformal intervals achieved 89.9% coverage for a 90% target. Under a "hard" shape shift, coverage dropped to 85.3% but was restored to 90% using the 30-label recalibration strategy.
  • Massive Speedup: In a resistive BOUT++ scenario scan, the trust layer screened 42 operating points in just 2 seconds. The equivalent physics simulations would have taken 12 hours. The system settled 43–98% of decisions confidently without needing the slow code.
  • OOD Success: The ensemble-spread alarm successfully flagged all three "unseen" tokamaks (JET, NSTX, ITER) as "Out of Range," preventing the surrogate from providing untrustworthy predictions on those machines.
Figure 4: The surrogate correctly flagging real-machine equilibria as "Out of Range."

Limitations and Future Work

  • Marginal vs. Conditional Guarantees: Conformal prediction guarantees coverage on average across the distribution. It cannot guarantee that a specific, individual prediction is correct, which is why routing to physics codes remains a necessary safety feature.
  • Exchangeability Assumption: The method assumes that the calibration data and test data are "exchangeable." While recalibration and weighting help, extreme shifts still pose a challenge.
  • Experimental Gap: This work compares surrogates to physics codes (MISHKA, BOUT++). The next critical step is validating these models against actual experimental data of ELM (Edge Localized Mode) onset in real tokamaks.
  • Ideal vs. Real Physics: Much of the large-scale testing was on ideal-MHD data. Real-world resistive MHD is more complex, and the authors note that cross-machine skill requires training on datasets with more realistic profile shapes.

Summary updated

Abstract

Machine-learning surrogates of linear peeling-ballooning stability now run in milliseconds, but they are mostly deployed as point estimators: they do not say how wrong they may be, or when a query lies outside what they were trained on. For a fusion company designing a new device, every equilibrium of interest is outside the training data of existing surrogates, so this is the case that matters. We describe a model-agnostic trust layer for stability surrogates, made of conformal error bars, out-of-distribution alarms that route doubtful cases to the physics code, and recalibration from a few simulations of a new design, and we test it with pre-registered hypotheses. On a large public dataset of JET-like equilibria with ideal-MHD growth rates, the error bars hold their nominal coverage in distribution and fall short under a shift in plasma shape; a few dozen simulations of the new shape, or routing the cases the alarms flag, restore it. On resistive BOUT++ surrogates, the bars stay calibrated on held-out runs, the alarm flags tokamaks absent from training, and a scenario scan is screened in seconds instead of hours. Across equilibria from the SPARC library and the DIII-D coil set, a surrogate trained on our own graded BOUT++ labels beats the baseline on unseen equilibria, while a surrogate trained elsewhere ranks the same cases backwards and its alarm declines them. Building our own neural-operator surrogate, the harness exposed two input defects whose fix closed its gap to a simple network. For every surrogate tested, the trust layer keeps coverage near target, with a bar width set by that surrogate's own error. Not every pre-registered hypothesis held, and we report those results and the limits of all of them.
BibTeX
@misc{vakhshouri2026trust,
  author = {Amin Vakhshouri},
  title = {{A trust layer for fast pedestal-stability surrogates: calibrated error bars, out-of-distribution alarms and few-shot recalibration}},
  year = {2026},
  eprint = {2611.06597},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06597},
}
Figure