TL;DR
The authors introduce RNN-Opt, a meta-learning approach that trains recurrent neural networks to act as sample-efficient black-box optimizers. By directly minimizing a discounted regret-based loss function and incorporating dynamic input normalization, RNN-Opt outperforms traditional and existing learned optimizers under strict evaluation budgets. Furthermore, an extension called RNN-Opt-DC successfully integrates domain constraints, making it highly effective for real-world industrial applications.
What Problem Does This Paper Address?
Black-box optimization (BBO) is crucial when dealing with complex, real-world systems where the underlying objective function lacks a closed-form analytical expression and gradient information is unavailable. Evaluating these systems (e.g., running a physical experiment or a heavy simulation) is often expensive and time-consuming, demanding optimization algorithms that can find near-optimal solutions within a very limited query budget.
While traditional methods like Bayesian Optimization or CMA-ES are standard, recent attention has shifted toward "learning to learn" (meta-learning) frameworks. Here, a neural network—typically a Recurrent Neural Network (RNN)—is trained on a distribution of synthetic functions to learn an efficient optimization strategy. However, existing meta-learning approaches suffer from key limitations:
- Suboptimal Loss Functions: They typically train models to maximize observed improvement at each step, which encourages greedy, short-sighted queries.
- Scale Sensitivity: Neural networks struggle when the range of the black-box function's output is unknown or varies wildly across applications.
- Lack of Constraint Handling: Real-world engineering problems are rarely unconstrained, yet standard learned optimizers lack mechanisms to respect parameter boundaries or domain constraints.
Methods
RNN-Opt models the optimization process using an LSTM-based architecture. Instead of deterministically predicting the next query point, the RNN outputs the parameters of a multivariate Gaussian distribution from which the next query point $x_{t+1}$ is sampled, promoting better exploration.
The core innovations of RNN-Opt include:
-
Regret-Based Loss Function: Instead of using Observed Improvement (OI), the authors train the network using a discounted regret loss function: $$L_R = \sum_{f_g \in \mathcal{F}} \sum_{t=2}^T \frac{1}{\gamma^t} \text{ReLU}(y_{opt} - \max_{i \le t} y_i)$$ This directly penalizes the model based on the distance between the true global optimum ($y_{opt}$) and the best value found so far. A discount factor ($\gamma < 1$) gives exponentially higher weight to later steps, acknowledging that early steps are naturally exploratory.
-
Incremental Normalization: To prevent the LSTM's activation functions (like sigmoid or tanh) from saturating when encountering unexpected function values, the authors introduce dynamic scaling. The raw function response $y_t$ is normalized on the fly using the running mean and variance of all observed responses before being fed back into the RNN.
- Handling Domain Constraints (RNN-Opt-DC): For constrained optimization, the authors introduce a penalty feedback loop. If a proposed query violates constraints, a penalty value $p_t$ is computed and fed into the RNN. Concurrently, a constraint loss term ($L_D$) is added to the total objective function, explicitly training the network to avoid invalid regions.
Datasets
The model is meta-trained and evaluated on several function sets:
- GMM-DF (Gaussian Mixture Model Density Functions): A diverse set of synthetically generated, non-convex differentiable functions used for both training and testing.
- Standard Mathematical Benchmarks: Classic, challenging optimization benchmarks including the Goldstein, Rosenbrock, Rastrigin, and Spherical functions (tested at dimensions $d=2$ and $d=6$).
- Industrial Constrained Optimization Problem: An anonymized, non-linear surrogate model of a real-world industrial process featuring 6 controllable input parameters subject to strict domain-knowledge constraints.
Key Insights and Ideas
- Supervised vs. Unsupervised Meta-Learning: Training with a known target ($y_{opt}$) via regret loss acts like a supervised learning setup, providing a much stronger and more stable directional gradient for the optimizer than the unsupervised-like nature of observed improvement.
- The Importance of Discounting: For regret-based optimization, setting $\gamma < 1$ is essential. Without it, the network is heavily penalized for its initial random guesses, which ruins its ability to converge effectively in later steps.
- Mitigating Network Saturation: Without incremental normalization, the RNN completely fails on functions with large output ranges. As shown in the paper's ablation studies, unnormalized inputs cause the network to saturate instantly, leading to flatlined performance.
Key Results
- Superiority over Baselines: Under a strict budget ($T = 20$ steps for $d=2$; $T = 60$ steps for $d=6$), RNN-Opt consistently achieved lower final regret compared to traditional methods (CMA-ES, Nelder-Mead) and the previous state-of-the-art learned optimizer (RNN-OI) across almost all benchmark functions.
- Constraint Adherence: RNN-Opt-DC successfully navigated constrained spaces, finding feasible, low-regret solutions much faster than baseline models that only accounted for constraints post-hoc during inference. It proved highly effective on the real-world industrial process benchmarks, rapidly adapting to parameter boundaries.
Limitations and Future Work
The primary limitation of RNN-Opt is its fixed input dimensionality; a model trained for a 2D optimization problem cannot inherently be applied to a 6D problem without retraining.
To address this, the authors point to future work involving scaling architectures that can ingest varying numbers of parameters—potentially by mapping high-dimensional spaces into a flexible latent space first. Additionally, extending RNN-Opt to handle multi-objective optimization problems remains an open and promising direction.
Summary updated