Meta-Learning for Black-box Optimization

Vishnu Tv, Pankaj Malhotra, Jyoti Narwariya, Lovekesh Vig, Gautam ShroffarXiv 2019 · arXiv

TL;DR

The authors introduce RNN-Opt, a meta-learning approach that trains recurrent neural networks to act as sample-efficient black-box optimizers. By directly minimizing a discounted regret-based loss function and incorporating dynamic input normalization, RNN-Opt outperforms traditional and existing learned optimizers under strict evaluation budgets. Furthermore, an extension called RNN-Opt-DC successfully integrates domain constraints, making it highly effective for real-world industrial applications.

What Problem Does This Paper Address?

Black-box optimization (BBO) is crucial when dealing with complex, real-world systems where the underlying objective function lacks a closed-form analytical expression and gradient information is unavailable. Evaluating these systems (e.g., running a physical experiment or a heavy simulation) is often expensive and time-consuming, demanding optimization algorithms that can find near-optimal solutions within a very limited query budget.

While traditional methods like Bayesian Optimization or CMA-ES are standard, recent attention has shifted toward "learning to learn" (meta-learning) frameworks. Here, a neural network—typically a Recurrent Neural Network (RNN)—is trained on a distribution of synthetic functions to learn an efficient optimization strategy. However, existing meta-learning approaches suffer from key limitations:

  1. Suboptimal Loss Functions: They typically train models to maximize observed improvement at each step, which encourages greedy, short-sighted queries.
  2. Scale Sensitivity: Neural networks struggle when the range of the black-box function's output is unknown or varies wildly across applications.
  3. Lack of Constraint Handling: Real-world engineering problems are rarely unconstrained, yet standard learned optimizers lack mechanisms to respect parameter boundaries or domain constraints.

Methods

RNN-Opt models the optimization process using an LSTM-based architecture. Instead of deterministically predicting the next query point, the RNN outputs the parameters of a multivariate Gaussian distribution from which the next query point $x_{t+1}$ is sampled, promoting better exploration.

The core innovations of RNN-Opt include:

  • Regret-Based Loss Function: Instead of using Observed Improvement (OI), the authors train the network using a discounted regret loss function: $$L_R = \sum_{f_g \in \mathcal{F}} \sum_{t=2}^T \frac{1}{\gamma^t} \text{ReLU}(y_{opt} - \max_{i \le t} y_i)$$ This directly penalizes the model based on the distance between the true global optimum ($y_{opt}$) and the best value found so far. A discount factor ($\gamma < 1$) gives exponentially higher weight to later steps, acknowledging that early steps are naturally exploratory.

  • Incremental Normalization: To prevent the LSTM's activation functions (like sigmoid or tanh) from saturating when encountering unexpected function values, the authors introduce dynamic scaling. The raw function response $y_t$ is normalized on the fly using the running mean and variance of all observed responses before being fed back into the RNN.

Fig. 1. Computation flow in RNN-Opt. During training, the functions fg are differen- tiable and obtained using Equation 12. Once trained, fg is replaced by the black-box function fb.
  • Handling Domain Constraints (RNN-Opt-DC): For constrained optimization, the authors introduce a penalty feedback loop. If a proposed query violates constraints, a penalty value $p_t$ is computed and fed into the RNN. Concurrently, a constraint loss term ($L_D$) is added to the total objective function, explicitly training the network to avoid invalid regions.
Fig. 3. Computation flow in RNN-Opt-DC. Here fg is the function to be optimized, and fp is used to compute the penalty pt. Further, if pt = 0, actual value of fg, i.e. yt is passed to the loss function and RNN, else yt is set to yt−1.

Datasets

The model is meta-trained and evaluated on several function sets:

  • GMM-DF (Gaussian Mixture Model Density Functions): A diverse set of synthetically generated, non-convex differentiable functions used for both training and testing.
  • Standard Mathematical Benchmarks: Classic, challenging optimization benchmarks including the Goldstein, Rosenbrock, Rastrigin, and Spherical functions (tested at dimensions $d=2$ and $d=6$).
  • Industrial Constrained Optimization Problem: An anonymized, non-linear surrogate model of a real-world industrial process featuring 6 controllable input parameters subject to strict domain-knowledge constraints.

Key Insights and Ideas

  • Supervised vs. Unsupervised Meta-Learning: Training with a known target ($y_{opt}$) via regret loss acts like a supervised learning setup, providing a much stronger and more stable directional gradient for the optimizer than the unsupervised-like nature of observed improvement.
  • The Importance of Discounting: For regret-based optimization, setting $\gamma < 1$ is essential. Without it, the network is heavily penalized for its initial random guesses, which ruins its ability to converge effectively in later steps.
  • Mitigating Network Saturation: Without incremental normalization, the RNN completely fails on functions with large output ranges. As shown in the paper's ablation studies, unnormalized inputs cause the network to saturate instantly, leading to flatlined performance.
Fig. 2. Effect of not using suitable scaling (incremental normalization in our case) of black-box function value during inference.

Key Results

  • Superiority over Baselines: Under a strict budget ($T = 20$ steps for $d=2$; $T = 60$ steps for $d=6$), RNN-Opt consistently achieved lower final regret compared to traditional methods (CMA-ES, Nelder-Mead) and the previous state-of-the-art learned optimizer (RNN-OI) across almost all benchmark functions.
  • Constraint Adherence: RNN-Opt-DC successfully navigated constrained spaces, finding feasible, low-regret solutions much faster than baseline models that only accounted for constraints post-hoc during inference. It proved highly effective on the real-world industrial process benchmarks, rapidly adapting to parameter boundaries.

Limitations and Future Work

The primary limitation of RNN-Opt is its fixed input dimensionality; a model trained for a 2D optimization problem cannot inherently be applied to a 6D problem without retraining.

To address this, the authors point to future work involving scaling architectures that can ingest varying numbers of parameters—potentially by mapping high-dimensional spaces into a flexible latent space first. Additionally, extending RNN-Opt to handle multi-objective optimization problems remains an open and promising direction.

Summary updated

Abstract

Recently, neural networks trained as optimizers under the "learning to learn" or meta-learning framework have been shown to be effective for a broad range of optimization tasks including derivative-free black-box function optimization. Recurrent neural networks (RNNs) trained to optimize a diverse set of synthetic non-convex differentiable functions via gradient descent have been effective at optimizing derivative-free black-box functions. In this work, we propose RNN-Opt: an approach for learning RNN-based optimizers for optimizing real-parameter single-objective continuous functions under limited budget constraints. Existing approaches utilize an observed improvement based meta-learning loss function for training such models. We propose training RNN-Opt by using synthetic non-convex functions with known (approximate) optimal values by directly using discounted regret as our meta-learning loss function. We hypothesize that a regret-based loss function mimics typical testing scenarios, and would therefore lead to better optimizers compared to optimizers trained only to propose queries that improve over previous queries. Further, RNN-Opt incorporates simple yet effective enhancements during training and inference procedures to deal with the following practical challenges: i) Unknown range of possible values for the black-box function to be optimized, and ii) Practical and domain-knowledge based constraints on the input parameters. We demonstrate the efficacy of RNN-Opt in comparison to existing methods on several synthetic as well as standard benchmark black-box functions along with an anonymized industrial constrained optimization problem.
BibTeX
@misc{tv2019meta,
  author = {Vishnu TV and Pankaj Malhotra and Jyoti Narwariya and Lovekesh Vig and Gautam Shroff},
  title = {{Meta-Learning for Black-box Optimization}},
  year = {2019},
  eprint = {1907.06901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.06901},
}

Loading related content...

Figure