Introduction
In the current landscape of deep learning, the performance of a model is often as dependent on the optimization algorithm as it is on the architecture itself. While hand-designed optimizers like Adam have become the industry standard, they remain notoriously sensitive to hyperparameter tuning—particularly the learning rate. A setting that yields state-of-the-art results on one dataset may lead to divergence on another.
Mnemosyne introduces a paradigm shift by replacing these rigid, hand-crafted rules with a learnable optimizer based on Transformers. By leveraging a novel "Compact Associative Memory" (CAM) and efficient spatio-temporal attention, Mnemosyne can optimize massive neural networks—including those with over 11 billion parameters—without requiring task-specific hyperparameter tuning.
The Challenge of Learning-to-Learn
The field of "Learning-to-Learn" (L2L) seeks to automate the optimization process by training a neural network to act as the optimizer. Historically, L2L systems have relied on Recurrent Neural Networks (RNNs) like LSTMs. However, LSTMs face two significant hurdles: they suffer from "catastrophic forgetting" over the long horizons required for model training, and they struggle to scale to the parameter counts of modern foundation models.
While standard Transformers could theoretically solve the memory issue, their quadratic computational complexity ($O(L^2)$) makes them prohibitively expensive when processing long sequences of gradients. Mnemosyne addresses these limitations through a suite of architectural innovations designed for efficiency and scalability.
The Mnemosyne Architecture
Mnemosyne is built on several key technical pillars that allow it to function as a high-performance optimizer:
1. Spatio-Temporal Low-Rank Attention
To overcome the quadratic bottleneck of standard attention, Mnemosyne utilizes Performers. By using random features to approximate the softmax kernel, the system achieves linear complexity ($O(L)$) relative to the sequence length. This allows the optimizer to maintain and process long optimization histories without a massive computational overhead.
2. Compact Associative Memory (CAM)
The temporal encoder in Mnemosyne functions as a fixed-size hidden state that acts as a differentiable dictionary. This Compact Associative Memory stores patterns of optimization trajectories and retrieves them to generate parameter updates. Theoretically, the authors demonstrate that CAM can store an exponential number of patterns relative to its dimensionality, explaining its resilience against the forgetting issues seen in LSTMs.
3. Topological Encodings and Hierarchical Pooling
Mnemosyne processes parameters using a tree-like structure based on the target model's architecture. It offers two modes of operation:
- Coordinate-wise: Treating every scalar parameter independently.
- Tensor-wise: Processing entire weight matrices as units, which significantly improves memory efficiency.
To handle massive tensors (e.g., $1024 \times 1024$ matrices), Mnemosyne employs a hierarchical encoder to compress long sequences of parameters into a manageable set of "meta-tokens."
Experimental Evaluation
The authors evaluated Mnemosyne's generalization capabilities across a broad spectrum of tasks, testing whether an optimizer trained on simple tasks could scale to complex, real-world models.
- Computer Vision: Fine-tuning Vision Transformers (ViT-Base, ViT-Large, and the 36-layer ViT-Huge) on datasets including ImageNet-2012, Places365, and Caltech-Birds-2011.
- Natural Language Processing: Pre-training BERT models (86M parameters) using Masked Language Modeling on Wikipedia and the Books dataset.
- Large-Scale Scaling: Soft prompt-tuning of the 11B+ parameter T5XXL model on the SuperGLUE benchmark.
Key Results and Insights
Mnemosyne demonstrates that the attention mechanisms which revolutionized NLP and Vision are equally potent when applied to the optimization process itself.
- Zero-Shot Generalization: Remarkably, Mnemosyne was meta-trained on simple MNIST tasks but successfully optimized ViT-Huge and T5 models on entirely different datasets without further adjustment. This suggests the optimizer learns fundamental principles of gradient descent rather than task-specific shortcuts.
- Implicit Hyperparameter Tuning: Mnemosyne effectively "learns" the optimal learning rate and momentum schedules. In head-to-head comparisons, it matched the performance of an Adam optimizer that had been meticulously hand-tuned, while remaining stable even in scenarios where Adam would typically fail due to poor hyperparameter choices.
- Scaling to 11 Billion Parameters: The system successfully handled soft prompt-tuning for T5XXL, a scale previously considered unreachable for most L2L systems. The "tensor-wise" variant provides space complexity comparable to standard first-order optimizers, making it a viable candidate for production-scale training.
Limitations and Future Work
Despite its success, Mnemosyne faces challenges in the meta-training phase. Training the optimizer itself is a complex task that requires a hybrid loss function—combining standard task loss with an imitation loss from an "expert" optimizer like Adam—to ensure stability. Additionally, while the tensor-wise mode is efficient, the coordinate-wise mode still carries a linear multiplicative constant that can be demanding for very deep temporal encoders.
Future research will likely focus on more sophisticated meta-training strategies and investigating whether attention-based optimizers can further reduce total training time by improving convergence rates on foundational models.
Summary updated