Abstract
In many engineering and scientific optimization tasks, the objective function is subject to multiple black-box constraints. Standard Constrained Bayesian Optimization (cBO) typically assumes these constraints are independent to simplify the calculation of the Probability of Feasibility (PoF). This paper investigates whether modeling the statistical dependence between constraints improves optimization performance. We introduce "Dep-PoF," a framework utilizing Multiple Output Gaussian Processes (MOGPs) and Expectation Propagation (EP) to account for constraint correlations. Our findings suggest that while modeling dependence is theoretically superior, the practical gains are often marginal, justifying the continued use of simpler independence assumptions in most scenarios.
Introduction
Constrained Bayesian Optimization (cBO) is a powerful tool for optimizing expensive-to-evaluate black-box functions under safety or physical constraints. A critical component of cBO is the surrogate model, which estimates the Probability of Feasibility (PoF)—the likelihood that a given point satisfies all constraints. Historically, researchers have treated multiple constraints as independent variables, calculating the joint PoF as the product of individual probabilities.
As shown in Table 1, the majority of existing methodologies either assume independence or do not explicitly model the joint distribution of constraints. However, in physical systems—such as chemical reactors where temperature and pressure are intrinsically linked—this assumption may be flawed. This research explores whether explicitly modeling these dependencies leads to faster convergence or better optima.
Methodology: Modeling Correlation with Dep-PoF
To move beyond the independence assumption, we propose Dep-PoF. This approach relies on two sophisticated technical pillars:
- Multiple Output Gaussian Processes (MOGPs): Instead of training separate Gaussian Processes for each constraint, MOGPs model all constraints simultaneously. This allows the surrogate to capture the cross-covariance between different outputs, effectively "learning" how constraints move together.
- Expectation Propagation (EP): Calculating the joint probability that multiple correlated variables fall within a feasible region is a high-dimensional integration problem that is computationally prohibitive. We employ Expectation Propagation to approximate these joint probabilities efficiently, making the dependent PoF calculation tractable for real-time optimization.
We integrated Dep-PoF into standard acquisition functions, specifically Constrained Expected Improvement (cEI) and Constrained Adaptive Sampling (cAS), to create their "Dependent" counterparts.
Experimental Evaluation
The proposed methods were tested against a wide array of benchmarks:
- Synthetic Challenges: Standard optimization problems (G4 through G10) with varying dimensions (2D to 10D).
- Real-World Engineering: Design problems including tension-compression strings, pressure vessels, welded beams, and speed reducers.
- Extreme Dependence Case: A toy example where a single constraint was intentionally repeated to simulate perfect correlation.
Key Insights and Results
The most striking result of this study is a "win for simplicity." In the vast majority of standard benchmarks and real-world engineering problems, modeling dependence yielded performance that was virtually indistinguishable from the independent model.
The Convergence to Independence
The authors provide a theoretical explanation for this: as the number of observations increases, the surrogate models (GPs) become increasingly accurate. Over time, the "Dependent PoF" naturally converges to the "Independent PoF." Consequently, any benefit from modeling correlation is a transient, early-stage phenomenon that quickly diminishes as more data is collected.
When Does Dependence Matter?
While the independence assumption is generally robust, two specific scenarios favored Dep-PoF:
- Extremely Small Feasible Regions: In problems like G7 and G10, where the feasible space is a tiny fraction of the total search space, modeling dependence helped the algorithm navigate the boundaries more effectively.
- Redundant Constraints: In the toy case with repeated constraints, the independent model "over-penalized" the PoF by multiplying the same probability multiple times, leading to overly conservative exploration. Dep-PoF correctly identified the redundancy.
Conclusion
For the practitioner, this research offers a reassuring conclusion: the standard assumption of constraint independence in Bayesian Optimization is remarkably effective. While modeling dependence via MOGPs and EP is technically sound, the significant computational overhead is rarely justified by the marginal performance gains. Modeling dependence should be reserved for niche cases involving extremely expensive experiments and highly restricted feasible regions where every percentage point of efficiency is critical.
Summary updated