EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization

Ondrej Bohdal, Yongxin Yang, Timothy M. Hospedales

TL;DR

EvoGrad is a novel meta-learning algorithm that replaces expensive second-order gradients with a lightweight, evolutionary-inspired estimator. By avoiding the need to store long computational graphs, it enables researchers to scale meta-learning to much larger neural architectures (like ResNet34) while significantly reducing training time and memory usage—often by 50% or more—without sacrificing accuracy.

The Bottleneck in Meta-Learning

Meta-learning, or "learning how to learn," is a powerful paradigm where a model doesn't just learn a task, but also learns the best way to learn that task (e.g., tuning its own hyperparameters, loss functions, or initializations).

However, standard gradient-based meta-learning (GBML) has a major bottleneck: bilevel optimization. To update a hyperparameter, you usually have to differentiate through the entire training process. This requires computing second-order derivatives (Hessians) and storing a massive computational graph for backpropagation.

In practice, this means meta-learning is often stuck using tiny models (like ResNet10) because larger ones simply won't fit in GPU memory. Even on a high-end 12GB GPU, the memory cost of "unrolling" the training loop can be several times larger than standard training.

Introducing EvoGrad

The authors introduce EvoGrad, which draws inspiration from evolutionary strategies to estimate "hypergradients" (the gradient of validation loss with respect to hyperparameters) more efficiently.

The "Putative" Evolutionary Update

EvoGrad's evolutionary update mechanism for estimating hypergradients without second-order derivatives.

Instead of performing a standard gradient descent step in the inner loop to estimate the hypergradient, EvoGrad executes the following steps:

  1. Perturb: It creates $K$ copies of the current model weights (shown as $\theta_1$ and $\theta_2$ in the figure above), adding a small amount of random noise ($\epsilon$) to each.
  2. Evaluate: It calculates the training loss ($\ell$) for each noisy copy using the meta-knowledge $\lambda$.
  3. Weight: It uses a softmax function (controlled by a temperature parameter) to assign "fitness" weights ($w_1, w_2$) to these copies based on their loss.
  4. Combine: It creates a "virtual" updated model ($\theta^*$) by taking a weighted average of the copies.
  5. Differentiate: It calculates the gradient of the validation loss ($\ell_V$) at this virtual state with respect to the hyperparameters ($\lambda$).

The Magic Trick: Because the "update" to the virtual model was just a weighted sum of random perturbations—not a gradient-based step—calculating the final gradient doesn't trigger second-order derivatives. This keeps the computational graph short and the memory footprint small. Crucially, the actual model training still uses standard gradient descent; the evolutionary part is used only to estimate the hypergradient.

Experimental Validation

The authors evaluated EvoGrad across diverse domains to prove its generality:

  • Computer Vision (Few-Shot Learning): CUB, Cars, Places, and Plantae datasets using the LFT-RelationNet architecture.
  • Computer Vision (Noisy Labels): CIFAR-10 and CIFAR-100, using Meta-Weight-Net to learn how to ignore mislabeled data.
  • NLP (Cross-Lingual): The WikiAnn dataset for named entity recognition (NER) across multiple low-resource languages.
  • Proof of Concept: MNIST (for a rotation transformation task) and a synthetic 1D optimization problem.

Key Insights and Results

  • Efficiency over Exactness: The authors found that even with just $K=2$ model copies, the hypergradient estimate is accurate enough to guide optimization. Mathematically, EvoGrad acts as a random projection of the high-dimensional validation gradient into a lower-dimensional space, explaining why it works even when the estimate is "noisy."
  • Breaking the Scaling Ceiling: Because EvoGrad doesn't require the memory-heavy "unrolling" of the inner loop, it allows meta-learning to move from ResNet10 to ResNet34 on standard hardware.
  • Memory Savings & Speedup: In cross-domain few-shot learning, EvoGrad reduced memory usage from 8.3GB to 4.8GB compared to the standard algorithm. Training time per epoch was nearly 2x faster in the same few-shot tasks and about 33% faster in noisy label experiments.
  • Accuracy: Despite being a "noisy" estimator, EvoGrad matched or slightly exceeded the accuracy of standard second-order methods across all benchmarks. On the WikiAnn NLP task, it achieved an average F1 score of 71.98% vs. 71.13% for the baseline.

Limitations and Future Directions

While EvoGrad offers massive efficiency gains, it has a few limitations:

  • Greediness: EvoGrad is a "greedy" optimizer; it only looks at the immediate next step of the model rather than the long-term training trajectory, which can sometimes lead to sub-optimal hyperparameter settings.
  • Stochasticity: The gradient estimate is inherently noisy. While $K=2$ worked for the tested problems, more complex tasks might require a larger population, increasing compute costs.
  • Hyper-Hyperparameters: EvoGrad introduces its own parameters, like the perturbation scale and softmax temperature, which may require tuning.

Future work may explore more sophisticated evolutionary strategies (like CMA-ES) for the inner loop to provide even better gradient estimates while maintaining these impressive memory and speed benefits.

Summary updated

Abstract

Gradient-based meta-learning and hyperparameter optimization have seen significant progress recently, enabling practical end-to-end training of neural networks together with many hyperparameters. Nevertheless, existing approaches are relatively expensive as they need to compute second-order derivatives and store a longer computational graph. This cost prevents scaling them to larger network architectures. We present EvoGrad, a new approach to meta-learning that draws upon evolutionary techniques to more efficiently compute hypergradients. EvoGrad estimates hypergradient with respect to hyperparameters without calculating second-order gradients, or storing a longer computational graph, leading to significant improvements in efficiency. We evaluate EvoGrad on three substantial recent meta-learning applications, namely cross-domain few-shot learning with feature-wise transformations, noisy label learning with Meta-Weight-Net and low-resource cross-lingual learning with meta representation transformation. The results show that EvoGrad significantly improves efficiency and enables scaling meta-learning to bigger architectures such as from ResNet10 to ResNet34.
BibTeX
@inproceedings{bohdal2021evograd,
  author = {Ondrej Bohdal and Yongxin Yang and Timothy M. Hospedales},
  title = {{EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bkVLXjRrV25},
}
Figure