TL;DR
This paper introduces HUB, a novel hybrid optimization strategy that combines the strengths of learned optimizers (like VeLO) with traditional hand-designed optimizers. By intelligently blending these approaches, HUB significantly improves generalization and robustness, achieving state-of-the-art performance across a wide range of tasks and architectures, particularly in challenging fine-tuning scenarios.
What Problem Does This Paper Address?
Learned optimizers, while powerful, often suffer from instability, poor generalization to unseen tasks or architectures, and difficulties in fine-tuning pre-trained models. These limitations hinder their widespread adoption. Existing methods to address these issues, such as fine-tuning the learned optimizer itself, are often computationally expensive and lack clear guidance. This paper aims to bridge this gap by proposing a method that leverages the benefits of both learned and hand-designed optimizers without requiring extensive retraining or complex hyperparameter tuning.
Methods
The core contribution is the Hybrid-Update-Based (HUB) optimization strategy. HUB works by dynamically blending the update signals from a learned optimizer (e.g., VeLO) and a hand-designed optimizer (e.g., Adam, Adamax). This blending is guided by a "hybrid reference weighting matrix" derived from the gradient magnitudes of each parameter.
- Core Idea: For parameters with small gradients, where learned optimizers excel at escaping local minima, HUB relies more on the learned optimizer. For parameters with large gradients, where hand-designed optimizers offer more stable descent, HUB leans on them.
- Continuous Prompt Tuning Analogy: The authors draw an analogy to prompt tuning in NLP/vision, where input modifications guide model behavior. In HUB, the output of the hand-designed optimizer acts as a "prompt" to guide the learned optimizer's updates.
- Variations: The paper also explores variations like "Invert Weighting HUB" (prioritizing hand-designed optimizers for fine-tuning) and "Block-wise HUB" (applying different weighting strategies to different network blocks).
Datasets
The paper evaluates HUB across a diverse set of tasks and datasets, including:
- Image Compression: HiP-CT 3D organ image dataset.
- Image Classification: CIFAR10, CIFAR100, Tiny-imagenet, and Imagenet.
- Trajectory Fitting & Lane-Keeping: Used with vanilla Neural ODEs and Liquid Time Constant (LTC) architectures.
- Autonomous Driving: DeepPiCar dataset.
Key Insights and Ideas
- Learned Optimizers Aren't Always Best: While learned optimizers are designed to be general, they can sometimes over-explore or make suboptimal decisions, especially in fine-tuning. HUB shows that a hybrid approach can be more robust.
- Gradient Magnitude as a Guide: Using the magnitude of gradients to dynamically switch between learned and hand-designed optimizers is an effective way to leverage their respective strengths. Small gradients benefit from exploration (learned), while large gradients benefit from stability (hand-designed).
- Fine-tuning is Different: The paper highlights that fine-tuning requires a different strategy than training from scratch. Inverting the weighting in HUB for fine-tuning tasks proves crucial for preserving pre-trained knowledge.
- Theoretical Justification: The authors provide a theoretical analysis showing how HUB can mitigate issues like gradient vanishing and exploding by intelligently combining optimizer behaviors.
Key Results
- Dominant Performance: HUB outperforms all competitors in 94% of the tested tasks, achieving superior testing performance.
- Robustness and Generalization: HUB demonstrates significant improvements in generalization across various model architectures (MLPs, CNNs, RNNs, Neural ODEs, Transformers) and tasks.
- Fine-tuning Success: HUB effectively addresses VeLO's poor performance in fine-tuning tasks, a significant limitation of learned optimizers.
- Specific Examples:
- On image compression tasks (MLP), HUB achieved higher PSNR than Adamax and VeLO.
- For Vision Transformer (ViT) image classification, HUB consistently achieved higher Top1/Top5 accuracy across CIFAR10, CIFAR100, and Tiny-imagenet.
- In RNN and Neural ODE tasks (LSTM/LTC), HUB significantly outperformed baseline optimizers and VeLO, especially in challenging scenarios where others failed or encountered numerical issues (NaNs).
Limitations and Future Work
- Computational Cost: While HUB aims to be efficient, the paper acknowledges that learned optimizers can have higher per-step overhead. Future work could focus on further optimizing this aspect.
- Generalizability of HUB: The paper suggests that HUB's principles can be applied to other learned optimizers beyond VeLO, which would be a valuable area for future research.
- Theoretical Exploration: While theoretical analysis is provided, further exploration into the theoretical underpinnings of HUB's stability and convergence properties could be beneficial.
- Block-wise HUB Potential: The paper introduces block-wise HUB as a promising direction for handling complex network structures with pre-trained and newly added components, suggesting further investigation into optimal weighting strategies for different blocks.
Summary updated