TL;DR
This paper introduces four new "Hessian-informed" trust-region filter (TRF) algorithms that significantly improve the efficiency and robustness of optimizing complex industrial processes. By incorporating information about the curvature of the problem (Hessian), these new methods drastically reduce the number of costly black-box evaluations and tuning parameters needed, outperforming existing methods.
What Problem Does This Paper Address?
Many industrial processes are modeled using "grey-box" approaches, which combine well-understood algebraic equations (glass-box) with components that are difficult or impossible to differentiate analytically (black-box). This hybrid nature poses a significant challenge for traditional optimization algorithms that rely on derivative information. While the established Trust-Region Filter (TRF) algorithm offers robustness, it often requires extensive parameter tuning and a large number of expensive black-box evaluations, making it inefficient for real-world applications. This paper aims to bridge this gap by developing more efficient and robust TRF variants that can better handle these grey-box problems.
Methods
The core contribution of this paper is the development of four new TRF algorithms (A1-A4) that leverage Hessian information. Unlike the baseline TRF (A0), these new variants incorporate projected positive definite Hessians. This is achieved through various strategies:
- Diagonal Loading (A1): A simple and computationally inexpensive method to perturb the Hessian to ensure it's positive definite.
- Spectral Projection (A2, A3, A4): These methods involve projecting the Hessian onto a positive definite space. A2 "clamps" negative eigenvalues to a small positive value, A3 replaces them with their absolute values, and A4 adaptively switches between A2 and A3 based on surrogate accuracy.
These Hessian-informed strategies enable automatic step scaling and reduce the need for manual parameter tuning. The algorithms also support a flexible surrogate modeling layer, allowing the use of low-fidelity (linear, quadratic) or high-fidelity (Taylor series, Gaussian Process) models for approximating the black-box components.
Datasets
The authors evaluated their algorithms on:
- A test set of 25 non-linear programming (NLP) problems with black-box components, compiled from simulation experiments and existing literature.
- Five engineering optimization problems: Himmelblau's problem, liquid-liquid extraction column, pressure vessel design, alkylation process, and spring design. These problems were formulated as constrained grey-box optimization problems.
Key Insights and Ideas
- Hessian information is crucial for grey-box optimization: Incorporating curvature information (Hessian) into TRF significantly improves performance by enabling more reliable and larger steps, reducing the need for restoration phases, and making the algorithm less sensitive to tuning parameters.
- Adaptive strategies offer a good balance: The adaptive switching strategy (A4) between clamped and absolute Hessian projections proved particularly effective, balancing curvature exploitation with step conservatism.
- High-fidelity surrogates are key: Gaussian Process (GP) and Taylor Series (TS) surrogates consistently outperformed lower-fidelity polynomial surrogates, especially when combined with Hessian-informed TRF variants. This highlights the importance of accurate local approximations for complex black-box functions.
- Reduced parameter tuning is a major win: The Hessian-based TRF variants require significantly fewer tuning parameters compared to the classical TRF, making them more user-friendly and robust.
Key Results
- The proposed Hessian-informed TRF variants (A1-A4) achieved up to an order-of-magnitude reduction in iterations and black-box evaluations compared to the classical TRF (A0).
- With high-fidelity surrogates (GP and TS), A1-A4 variants achieved 100% success rate on the benchmark problems, solving them in fewer than six iterations.
- Low-fidelity surrogates (linear, quadratic, simplified quadratic) achieved 72-84% success when paired with the best variants.
- The Hessian-based variants demonstrated reduced sensitivity to tuning parameters, with some requiring minimal or no tuning.
- In engineering case studies, A1-A4 methods often reached global minima that traditional solvers failed to identify.
- The performance profiles showed that A1 and A4 variants offered the most uniform and improved performance across most surrogate types.
Limitations and Future Work
- Scalability of GP surrogates: The paper notes that GP surrogate scalability can be a concern in high-dimensional spaces or with many data points, potentially becoming computationally burdensome.
- TS surrogate limitations: Taylor Series surrogates assume smoothness and derivative availability, which might not hold for noisy or non-differentiable black-box functions.
- Multiple trust-regions/sampling-regions: The paper suggests that for highly heterogeneous black-box process units, converging to a unified optimum might benefit from multiple trust-regions or sampling-regions.
- Future work: The authors suggest exploring gradient-enhanced GPs, hybridised neural-GP frameworks, and adaptively switching between TS and GP surrogates. Further research is also needed to assess performance on truly large-scale, high-dimensional, or highly constrained grey-box systems.
Summary updated