Abstract
In this work we introduce the stochastic nonlinear constrained derivative-free optimization method (S)NOWPAC (Stochastic Nonlinear Optimization With Path-Augmented Constraints). The method extends the derivative-free optimizer NOWPAC to be applicable for optimization under uncertainty. It is based on a trust-region framework, utilizing local fully quadratic surrogate models combined with Gaussian process surrogates to mitigate the noise in the objective function and constraint evaluations. We show the performance of our algorithm on a variety of robust optimization problems from the CUTEst benchmark suite by comparing to other state-of-the-art optimization methods. Although we focus on robust optimization benchmark problems to demonstrate (S)NOWPAC's capabilities, the optimizer can be applied to a broad range of applications in nonlinear constrained stochastic optimization.
Introduction
Modern engineering design often relies on complex "black-box" simulations where the underlying mathematical structure is inaccessible. In scenarios such as optimizing wind turbine yaw angles for maximum power output, researchers face two primary obstacles: the absence of analytical gradients and the presence of stochastic noise. Standard deterministic solvers frequently fail when confronted with the inherent variability of real-world data, while global methods like Bayesian Optimization often struggle to scale as the number of design variables increases.
To address these challenges, we present (S)NOWPAC (Stochastic Nonlinear Optimization With Path-Augmented Constraints), a derivative-free optimization framework designed for problems where both the objective function and constraints are corrupted by noise. By integrating a local trust-region approach with Gaussian Process (GP) smoothing, (S)NOWPAC provides a scalable, noise-resilient solution for local optimization.
Methodology
(S)NOWPAC builds upon the deterministic NOWPAC solver, introducing three critical enhancements to handle uncertainty:
1. Gaussian Process (GP) Smoothing
Rather than treating noisy simulation outputs as absolute truths, (S)NOWPAC employs GPs to "denoise" the data. By leveraging information from previously evaluated points, the algorithm constructs a smoother estimate of the objective and constraints, effectively filtering out high-frequency noise that would otherwise mislead the optimizer.
2. Noise-Adapted Trust-Region Management
In a standard trust-region framework, the solver seeks improvements within a local neighborhood. (S)NOWPAC dynamically adjusts the size of this region based on estimated noise levels. This prevents the trust region from shrinking prematurely in high-noise environments, ensuring that the local model remains "fully linear" and statistically reliable.
3. Feasibility Restoration Mode
Stochastic noise can lead a solver to inadvertently violate constraints. (S)NOWPAC includes a dedicated operational mode to handle such instances, steering the design back into the feasible region before resuming the primary optimization process.
Experimental Evaluation
The performance of (S)NOWPAC was evaluated using the Schittkowski benchmark set from the CUTEst suite. To simulate real-world uncertainty, artificial uniform noise was added to the deterministic objective functions and constraints. The test suite encompassed problems with 2 to 16 design variables and 1 to 10 constraints.
(S)NOWPAC was compared against several established methods, including COBYLA, NOMAD, and constrained Bayesian Optimization (cBO).
Key Insights
A central innovation of (S)NOWPAC is the local application of GPs. While traditional Bayesian Optimization uses GPs for global search—a computationally expensive task—(S)NOWPAC uses them locally to smooth the trust-region steps. This approach allows the method to "borrow" information from nearby points, reducing variance without requiring a massive increase in expensive simulation samples.
Furthermore, the method explicitly couples structural error (the accuracy of the quadratic model) with sampling error (the noise). By ensuring the trust-region radius does not fall below the "noise floor," (S)NOWPAC maintains stability where deterministic solvers typically stall.
Results and Discussion
The comparative analysis revealed several key advantages:
- Reliability: (S)NOWPAC solved a higher percentage of test problems within the allocated simulation budget than its competitors.
- Accuracy: In high-noise scenarios, (S)NOWPAC consistently converged closer to the true optimum, whereas solvers like COBYLA often failed to satisfy constraints or became trapped by noise.
- Scalability: While cBO performed competitively on low-dimensional problems (2-4 variables), its performance degraded significantly as dimensionality increased. (S)NOWPAC maintained its efficiency across more complex problem sets.
Conclusion and Future Work
(S)NOWPAC represents a significant advancement in derivative-free stochastic optimization. While highly effective, future development will focus on reducing the computational overhead of GP calculations through sparse approximations (e.g., SoR or FITC) and establishing formal proofs of convergence to first-order critical points. Additionally, exploring kernel selection strategies will further enhance the method's robustness across diverse non-stationary problems.
Summary updated