Abstract
Many cases exist in which a black-box function $f$ with high evaluation cost depends on two types of variables $\bm x$ and $\bm w$, where $\bm x$ is a controllable design variable and $\bm w$ are uncontrollable environmental variables that have random variation following a certain distribution $P$. In such cases, an important task is to find the range of design variables $\bm x$ such that the function $f(\bm x, \bm w)$ has the desired properties by incorporating the random variation of the environmental variables $\bm w$. A natural measure of robustness is the probability that $f(\bm x, \bm w)$ exceeds a given threshold $h$, which is known as the probability threshold robustness (PTR) measure. However, this measure cannot be correctly evaluated when the distribution $P$ is unknown. In this study, we addressed this problem by considering the distributionally robust PTR (DRPTR) measure, which considers the worst-case PTR within given candidate distributions. We propose a theoretically grounded and computationally efficient active learning method for identifying the reliable set where the DRPTR measure exceeds a desired probability $\alpha$.
Introduction
In engineering and manufacturing, design variables ($x$) are often subject to environmental variables ($w$) that fluctuate randomly. A common goal is Level-Set Estimation (LSE): identifying the set of design parameters where system performance $f(x, w)$ exceeds a threshold $h$ with high probability.
Standard approaches rely on the Probability Threshold Robustness (PTR) measure, assuming a known distribution for $w$. However, in practice, this distribution is often unknown or non-stationary. Relying on a single estimated distribution can lead to overconfident, unsafe designs—a phenomenon known as the "optimizer's curse." This paper introduces the Distributionally Robust PTR (DRPTR) problem, which seeks design regions that remain safe under the worst-case distribution within a set of plausible candidates.
Methodology
The proposed active learning framework aims to solve the DRPTR problem with minimal evaluations of the expensive black-box function $f(x, w)$.
Surrogate Modeling and Robust Measures
The method employs a Gaussian Process (GP) to model $f(x, w)$. The DRPTR measure $F(x)$ is defined as the minimum probability of success across a family of candidate distributions $\mathcal{A}$, typically defined by a distance metric (e.g., $L_1$ or $L_2$ norm) from a reference distribution. Because $F(x)$ is a complex transformation of the GP, the authors derive specific credible intervals to guide the active learning process.
Acquisition Functions and Optimization
Two new acquisition functions, $a_t^{(1)}$ and $a_t^{(2)}$, are proposed based on the Maximum Improvement in Level-set Estimation (MILE) strategy. These functions select the next $(x, w)$ pair by predicting which point will most significantly reduce uncertainty regarding the robust level set.
To overcome the computational burden of nested optimization and heavy sampling, the authors provide mathematical lemmas that transform these acquisition functions into simple linear or second-order cone programming problems. This optimization achieves speedups of up to 2,000x compared to naive implementations; for instance, a trial on the Booth function was reduced from 138,000 seconds to just 71 seconds.
Experimental Evaluation
The framework was evaluated using standard synthetic benchmarks (Booth, Matyas, McCormick, and Styblinski-Tang) and a real-world SIR model for infection control.
As illustrated in the figures above, the proposed methods (solid lines) reach high F-scores significantly faster than standard baselines like Uncertainty Sampling (US) or Random sampling (dashed lines). This indicates a more efficient identification of the robust level set. Furthermore, the DRPTR approach successfully identified "safe" regions that remained valid even when the actual environmental distribution deviated from the initial reference.
Conclusion and Future Work
This work provides a theoretically grounded method for robust level-set estimation under distributional uncertainty. By combining GP surrogates with efficient optimization lemmas, the authors enable practical application of distributionally robust optimization to expensive black-box functions. Future work includes extending the framework from discrete grids to continuous, high-dimensional spaces and addressing the cubic scaling limitations of Gaussian Processes.
Summary updated