Trust‐region filter algorithms utilizing Hessian information for gray‐box optimization

Gul Hameed, Tao Chen, Ehecatl Antonio del Rio‐Chanona, Lorenz T. Biegler, Michael ShortAIChE 2026 · DOI

TL;DR

This paper introduces four new "Hessian-informed" trust-region filter (TRF) algorithms that significantly improve the efficiency and robustness of optimizing complex industrial processes. By incorporating information about the curvature of the problem (Hessian), these new methods drastically reduce the number of costly black-box evaluations and tuning parameters needed, outperforming existing methods.

What Problem Does This Paper Address?

Many industrial processes are modeled using "grey-box" approaches, which combine well-understood algebraic equations (glass-box) with components that are difficult or impossible to differentiate analytically (black-box). This hybrid nature poses a significant challenge for traditional optimization algorithms that rely on derivative information. While the established Trust-Region Filter (TRF) algorithm offers robustness, it often requires extensive parameter tuning and a large number of expensive black-box evaluations, making it inefficient for real-world applications. This paper aims to bridge this gap by developing more efficient and robust TRF variants that can better handle these grey-box problems.

Methods

The core contribution of this paper is the development of four new TRF algorithms (A1-A4) that leverage Hessian information. Unlike the baseline TRF (A0), these new variants incorporate projected positive definite Hessians. This is achieved through various strategies:

  • Diagonal Loading (A1): A simple and computationally inexpensive method to perturb the Hessian to ensure it's positive definite.
  • Spectral Projection (A2, A3, A4): These methods involve projecting the Hessian onto a positive definite space. A2 "clamps" negative eigenvalues to a small positive value, A3 replaces them with their absolute values, and A4 adaptively switches between A2 and A3 based on surrogate accuracy.

These Hessian-informed strategies enable automatic step scaling and reduce the need for manual parameter tuning. The algorithms also support a flexible surrogate modeling layer, allowing the use of low-fidelity (linear, quadratic) or high-fidelity (Taylor series, Gaussian Process) models for approximating the black-box components.

Datasets

The authors evaluated their algorithms on:

  • A test set of 25 non-linear programming (NLP) problems with black-box components, compiled from simulation experiments and existing literature.
  • Five engineering optimization problems: Himmelblau's problem, liquid-liquid extraction column, pressure vessel design, alkylation process, and spring design. These problems were formulated as constrained grey-box optimization problems.

Key Insights and Ideas

  • Hessian information is crucial for grey-box optimization: Incorporating curvature information (Hessian) into TRF significantly improves performance by enabling more reliable and larger steps, reducing the need for restoration phases, and making the algorithm less sensitive to tuning parameters.
  • Adaptive strategies offer a good balance: The adaptive switching strategy (A4) between clamped and absolute Hessian projections proved particularly effective, balancing curvature exploitation with step conservatism.
  • High-fidelity surrogates are key: Gaussian Process (GP) and Taylor Series (TS) surrogates consistently outperformed lower-fidelity polynomial surrogates, especially when combined with Hessian-informed TRF variants. This highlights the importance of accurate local approximations for complex black-box functions.
  • Reduced parameter tuning is a major win: The Hessian-based TRF variants require significantly fewer tuning parameters compared to the classical TRF, making them more user-friendly and robust.

Key Results

  • The proposed Hessian-informed TRF variants (A1-A4) achieved up to an order-of-magnitude reduction in iterations and black-box evaluations compared to the classical TRF (A0).
  • With high-fidelity surrogates (GP and TS), A1-A4 variants achieved 100% success rate on the benchmark problems, solving them in fewer than six iterations.
  • Low-fidelity surrogates (linear, quadratic, simplified quadratic) achieved 72-84% success when paired with the best variants.
  • The Hessian-based variants demonstrated reduced sensitivity to tuning parameters, with some requiring minimal or no tuning.
  • In engineering case studies, A1-A4 methods often reached global minima that traditional solvers failed to identify.
  • The performance profiles showed that A1 and A4 variants offered the most uniform and improved performance across most surrogate types.

Limitations and Future Work

  • Scalability of GP surrogates: The paper notes that GP surrogate scalability can be a concern in high-dimensional spaces or with many data points, potentially becoming computationally burdensome.
  • TS surrogate limitations: Taylor Series surrogates assume smoothness and derivative availability, which might not hold for noisy or non-differentiable black-box functions.
  • Multiple trust-regions/sampling-regions: The paper suggests that for highly heterogeneous black-box process units, converging to a unified optimum might benefit from multiple trust-regions or sampling-regions.
  • Future work: The authors suggest exploring gradient-enhanced GPs, hybridised neural-GP frameworks, and adaptively switching between TS and GP surrogates. Further research is also needed to assess performance on truly large-scale, high-dimensional, or highly constrained grey-box systems.

Summary updated

Abstract

Abstract Optimizing industrial processes often involves gray‐box models that couple algebraic glass‐box equations with black‐box components lacking analytic derivatives. Such systems challenge derivative‐based solvers. The classical trust‐region filter (TRF) algorithm provides a robust framework but requires extensive parameter tuning and numerous black‐box evaluations. This work introduces four Hessian‐informed TRF variants that use projected positive definite Hessians for automatic step scaling and minimal tuning, combined with both low‐fidelity (linear, quadratic) and high‐fidelity (Taylor series, Gaussian process) surrogates for local black‐box approximation. Tested on 25 gray‐box benchmarks and five engineering case studies, the new variants achieved up to order‐of‐magnitude reductions in iterations and black‐box evaluations, with reduced sensitivity to tuning parameters relative to the classical TRF algorithm. High‐fidelity surrogates solved 92%–100% of problems, compared with 72%–84% for low‐fidelity surrogates. The developed TRF methods also outperformed classical derivative‐free optimization solvers. Results show that new variants offer robust, scalable alternatives for gray‐box optimization.
BibTeX
@inproceedings{hameed2026trustregion,
  author = {Gul Hameed and Tao Chen and Ehecatl Antonio del Rio‐Chanona and Lorenz T. Biegler and Michael Short},
  title = {{Trust‐region filter algorithms utilizing Hessian information for gray‐box optimization}},
  booktitle = {AIChE},
  year = {2026},
  doi = {10.1002/aic.70236},
}
Figure