Bayesian Experimental Design for Finding Reliable Level Set Under Input Uncertainty

Shogo Iwazaki, Yu Inatsu, Ichiro Takeuchi

Abstract

In the manufacturing industry, identifying the range of input conditions under which a machine operates properly often requires expensive testing. Because it is difficult to accurately control input conditions during actual usage, machine performance must be guaranteed by incorporating potential variations. We formulate this scenario as an Input Uncertain Reliable Level Set Estimation (IU-rLSE) problem. The goal is to identify input ranges where outputs meet a desired threshold with high probability under uncertainty. We propose an efficient active learning method, provide theoretical convergence analysis, and demonstrate its performance on synthetic and real-world data.

Introduction

In high-stakes engineering, the "Level Set" represents the range of inputs (e.g., temperature, pressure) where a system performs correctly. Traditional Level Set Estimation (LSE) assumes precise control over these inputs. However, in real-world manufacturing, inputs often fluctuate. A setting intended to be 100°C might vary between 98°C and 102°C. If an algorithm ignores this noise, it may select operating points that are safe on average but fail frequently due to these small variations.

The IU-rLSE Framework

To address this, we introduce IU-rLSE. This framework identifies regions where the probability of success remains high even when accounting for input uncertainty.

Comparison of standard LSE (a) and IU-rLSE (b, c). Standard LSE may classify points as safe based on mean performance, whereas IU-rLSE accounts for the input uncertainty distribution to ensure high-probability safety.

As shown in the figure above, standard LSE (a) might identify points as "lower than threshold $h$" without considering the risk posed by nearby unsafe regions. IU-rLSE explicitly models the input uncertainty (the distribution curves at the bottom of panels b and c). A point is only classified as reliable (c) if the output remains below $h$ with high probability across the entire noise distribution.

Methodology

The proposed algorithm uses Gaussian Process (GP) modeling to create a probabilistic map of the unknown function. The core innovation lies in the acquisition function:

  • Reliability Metric: Instead of mean performance, we optimize for the probability that the output stays within safe bounds.
  • Computational Efficiency: Calculating expected improvement under uncertainty typically involves complex triple integrals. We developed a mathematical approximation that reduces this to a single integral, making the method practical for real-time experimental design.
  • Adaptive Noise Estimation: The method can estimate unknown input noise parameters on the fly during the testing process.

Experimental Evaluation

We evaluated IU-rLSE against standard benchmarks and the Combined Cycle Power Plant (CCPP) dataset.

  • Superior Accuracy: IU-rLSE achieved higher F1-scores than traditional LSE methods like Straddle or MILE across various noise profiles (Gaussian and Gamma).
  • Sample Efficiency: On real-world power plant data, the method identified reliable operating regions significantly faster, requiring fewer expensive data points.
  • Theoretical Guarantees: We proved that the algorithm is mathematically guaranteed to converge to the correct reliable set over time.

Conclusion

IU-rLSE provides a robust framework for identifying safe operating regions in the presence of uncontrollable input noise. By combining high-probability safety guarantees with a computationally efficient active learning strategy, it offers a practical tool for modern manufacturing and engineering optimization.

Summary updated

Abstract

In the manufacturing industry, it is often necessary to repeat expensive operational testing of machine in order to identify the range of input conditions under which the machine operates properly. Since it is often difficult to accurately control the input conditions during the actual usage of the machine, there is a need to guarantee the performance of the machine after properly incorporating the possible variation in the input conditions. In this paper, we formulate this practical manufacturing scenario as an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Input Uncertain Reliable Level Set Estimation (IU-rLSE)</i> problem, and provide an efficient algorithm for solving it. The goal of IU-rLSE is to identify the input range in which the outputs smaller/greater than a desired threshold can be obtained with high probability when the input uncertainty is properly taken into consideration. We propose an active learning method to solve the IU-rLSE problem efficiently, theoretically analyze its accuracy and convergence, and illustrate its empirical performance through numerical experiments on artificial and real data.
BibTeX
@inproceedings{iwazaki2020bayesian,
  author = {Shogo Iwazaki and Yu Inatsu and Ichiro Takeuchi},
  title = {{Bayesian Experimental Design for Finding Reliable Level Set Under Input Uncertainty}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=nn_tg63lAv},
}
Figure