EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization

Ondrej Bohdal, Yongxin Yang, Timothy M. Hospedales

Abstract

Gradient-based meta-learning and hyperparameter optimization have seen significant progress recently, enabling practical end-to-end training of neural networks together with many hyperparameters. Nevertheless, existing approaches are relatively expensive as they need to compute second-order derivatives and store a longer computational graph. This cost prevents scaling them to larger network architectures. We present EvoGrad, a new approach to meta-learning that draws upon evolutionary techniques to more efficiently compute hypergradients. EvoGrad estimates hypergradient with respect to hyperparameters without calculating second-order gradients, or storing a longer computational graph, leading to significant improvements in efficiency. We evaluate EvoGrad on three substantial recent meta-learning applications, namely cross-domain few-shot learning with feature-wise transformations, noisy label learning with Meta-Weight-Net and low-resource cross-lingual learning with meta representation transformation. The results show that EvoGrad significantly improves efficiency and enables scaling meta-learning to bigger architectures such as from ResNet10 to ResNet34.

Introduction

Meta-learning, or "learning to learn," involves an outer loop that tunes hyperparameters (meta-parameters) and an inner loop that trains the actual model. Most current gradient-based methods rely on differentiating through the inner loop's update step to find the "hypergradient." This process typically requires second-order derivatives (Hessians) and forces the system to store the entire computational graph of the inner loop in memory. For large-scale models, this leads to memory exhaustion and excessive execution time. Consequently, many powerful meta-learning techniques have been restricted to small backbones like ResNet10, unable to scale to real-world, large-scale applications.

The EvoGrad Framework

EvoGrad addresses these bottlenecks by drawing inspiration from evolutionary strategies to estimate hypergradients using only first-order gradients. By decoupling the hypergradient from the inner-loop graph, the memory cost of meta-learning becomes nearly identical to standard training.

The EvoGrad computational flow for hypergradient estimation.

The core algorithm works as follows:

  1. Perturbation: Take the current model parameters and create a population of $K$ slightly different versions by adding random noise.
  2. Fitness Evaluation: Calculate the training loss for each version.
  3. Weighting: Assign weights to these versions using a softmax function based on their performance (lower loss = higher weight).
  4. Putative Update: Create a "virtual" updated model by taking a weighted average of the population.
  5. Hypergradient Estimation: Compute the gradient of the validation loss with respect to the meta-parameters by backpropagating through this virtual model.

Crucially, the actual training of the model still uses standard, efficient gradient descent. The evolutionary population is used only to estimate the direction for the hyperparameter update.

Experimental Results and Analysis

The authors evaluated EvoGrad across a diverse set of benchmarks, including computer vision (Few-Shot learning on CUB-200, Cars-196; Noisy Labels on CIFAR) and NLP (WikiAnn for cross-lingual NER).

Scaling and Efficiency

EvoGrad successfully scaled the "Learned Feature-wise Transformation" (LFT) method from a ResNet10 backbone to a ResNet34. Standard second-order approaches failed on a 12GB GPU due to memory limits. In experiments with Meta-Weight-Net, EvoGrad saved over 50% of GPU memory and reduced training time per epoch by approximately 33%.

Accuracy and Estimation Quality

Across all tasks, EvoGrad matched or slightly exceeded the accuracy of standard, more expensive meta-learning algorithms. A key insight is that EvoGrad essentially performs a random projection of the high-dimensional gradient into a lower-dimensional space, which remains effective even with very small populations (e.g., $K=2$).

Comparison of EvoGrad hypergradient estimates against ground truth for varying population sizes.

As shown in the figure above, while the hypergradient estimate is noisier than an exact second-order calculation for small populations, it tracks the ground truth trend well. The authors show that this noise does not hurt final performance and can even provide regularization that leads to better generalization.

Conclusion and Future Work

EvoGrad provides a practical solution for scaling meta-learning to deeper networks by eliminating the need for second-order derivatives and long computational graphs. While the method is "greedy"—optimizing for the immediate next step—and introduces its own hyper-hyperparameters (noise scale and temperature), its efficiency gains are substantial. Future directions include exploring more advanced evolutionary strategies, such as CMA-ES, to further improve gradient estimates.

Summary updated

Abstract

Gradient-based meta-learning and hyperparameter optimization have seen significant progress recently, enabling practical end-to-end training of neural networks together with many hyperparameters. Nevertheless, existing approaches are relatively expensive as they need to compute second-order derivatives and store a longer computational graph. This cost prevents scaling them to larger network architectures. We present EvoGrad, a new approach to meta-learning that draws upon evolutionary techniques to more efficiently compute hypergradients. EvoGrad estimates hypergradient with respect to hyperparameters without calculating second-order gradients, or storing a longer computational graph, leading to significant improvements in efficiency. We evaluate EvoGrad on three substantial recent meta-learning applications, namely cross-domain few-shot learning with feature-wise transformations, noisy label learning with Meta-Weight-Net and low-resource cross-lingual learning with meta representation transformation. The results show that EvoGrad significantly improves efficiency and enables scaling meta-learning to bigger architectures such as from ResNet10 to ResNet34.
BibTeX
@inproceedings{bohdal2021evograd,
  author = {Ondrej Bohdal and Yongxin Yang and Timothy M. Hospedales},
  title = {{EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bkVLXjRrV25},
}
Figure