Abstract
Model Agnostic Meta-Learning (MAML) is widely used to find a good initialization for a family of tasks. Despite its success, a critical challenge in MAML is to calculate the gradient w.r.t. the initialization over long inner-loop optimization paths. Standard MAML suffers from memory explosion and numerical instability when the number of gradient steps increases. We propose A-MAML, which leverages the adjoint method to compute gradients by treating the inner-loop optimization as an Ordinary Differential Equation (ODE). This approach enables learning from long training trajectories with constant memory overhead relative to the number of steps.
Introduction
Model-Agnostic Meta-Learning (MAML) has emerged as a cornerstone of few-shot learning, aiming to find a parameter initialization $\theta$ that can quickly adapt to new tasks. However, a significant bottleneck exists: the "inner loop" optimization. To update $\theta$, one must backpropagate through the entire optimization trajectory. As the number of gradient descent steps increases, the computation graph grows linearly, leading to prohibitive memory costs and numerical instability. Consequently, practical applications of MAML are often restricted to very few steps (e.g., 1–5), which prevents the model from learning initializations optimized for full convergence. Existing workarounds, such as First-Order MAML (FOMAML), often sacrifice accuracy, while others like Implicit MAML (iMAML) require restrictive mathematical assumptions.
Methodology: A-MAML
The proposed A-MAML framework addresses these limitations by reinterpreting the discrete gradient descent steps of the inner loop as a continuous-time dynamical system. Specifically, the inner-loop training is viewed as solving a forward Ordinary Differential Equation (ODE):
$$\frac{du}{dt} = -\nabla L(u)$$
where $u$ represents the model parameters and $L$ is the loss function.
To compute the gradient of the final validation loss with respect to the initialization $\theta$ without storing the entire forward trajectory, A-MAML employs the adjoint method. This involves solving a companion "adjoint" ODE backward in time. To maintain numerical stability during the backward pass—a common challenge in ODE solvers—the authors implement a second-order "modified Euler" solver and track states during the forward pass. This ensures that memory usage remains decoupled from the number of inner-loop steps, scaling only with the number of model parameters rather than the trajectory length.
Experimental Evaluation
The authors evaluated A-MAML across synthetic regression (Alpine, CosMixture), collaborative filtering (Jester, MovieLens), and image classification (Omniglot, Mini-ImageNet).
Memory and Computational Efficiency
A key advantage of A-MAML is its memory footprint. While standard MAML quickly exhausts GPU memory as the number of gradient descent (GD) steps increases, A-MAML maintains nearly constant memory usage.
As shown in the experimental results, standard MAML encounters out-of-memory (OOM) errors on a 24GB GPU at approximately 200 steps. In contrast, A-MAML scales to 500 steps using significantly less memory. Furthermore, the time per meta-update for A-MAML remains competitive with efficient first-order methods like Reptile and FOMAML, while providing the accuracy of higher-order methods.
Predictive Performance
A-MAML's ability to handle long trajectories translates to improved accuracy. On the MovieLens-1M task (20-shot), A-MAML achieved a normalized RMSE of 0.035, significantly outperforming standard MAML (0.059) and iMAML (0.052). On the Omniglot benchmark, A-MAML reached an accuracy of 96.79%, surpassing MAML, Reptile, and various iMAML variants. These results suggest that optimizing for longer trajectories allows the model to find more robust initializations.
Conclusion and Future Work
A-MAML provides a scalable solution for meta-learning by leveraging adjoint methods to bypass the memory constraints of unrolled computation graphs. By treating optimization as a continuous process, it enables the use of hundreds of inner-loop steps, which was previously computationally infeasible.
While the method assumes the optimization process can be modeled as a continuous ODE, it demonstrates robust performance across diverse domains. Future research may explore adapting this method for optimizers with discrete internal states or extending it to "conditional meta-learning," where the initialization is conditioned on task-specific side information.
Summary updated