HUB: Guiding Learned Optimizers with Continuous Prompt Tuning

Tiejun HuangarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces HUB, a novel hybrid optimization strategy that combines the strengths of learned optimizers (like VeLO) with traditional hand-designed optimizers. By intelligently blending these approaches, HUB significantly improves generalization and robustness, achieving state-of-the-art performance across a wide range of tasks and architectures, particularly in challenging fine-tuning scenarios.

What Problem Does This Paper Address?

Learned optimizers, while powerful, often suffer from instability, poor generalization to unseen tasks or architectures, and difficulties in fine-tuning pre-trained models. These limitations hinder their widespread adoption. Existing methods to address these issues, such as fine-tuning the learned optimizer itself, are often computationally expensive and lack clear guidance. This paper aims to bridge this gap by proposing a method that leverages the benefits of both learned and hand-designed optimizers without requiring extensive retraining or complex hyperparameter tuning.

Methods

The core contribution is the Hybrid-Update-Based (HUB) optimization strategy. HUB works by dynamically blending the update signals from a learned optimizer (e.g., VeLO) and a hand-designed optimizer (e.g., Adam, Adamax). This blending is guided by a "hybrid reference weighting matrix" derived from the gradient magnitudes of each parameter.

  • Core Idea: For parameters with small gradients, where learned optimizers excel at escaping local minima, HUB relies more on the learned optimizer. For parameters with large gradients, where hand-designed optimizers offer more stable descent, HUB leans on them.
  • Continuous Prompt Tuning Analogy: The authors draw an analogy to prompt tuning in NLP/vision, where input modifications guide model behavior. In HUB, the output of the hand-designed optimizer acts as a "prompt" to guide the learned optimizer's updates.
  • Variations: The paper also explores variations like "Invert Weighting HUB" (prioritizing hand-designed optimizers for fine-tuning) and "Block-wise HUB" (applying different weighting strategies to different network blocks).

Datasets

The paper evaluates HUB across a diverse set of tasks and datasets, including:

  • Image Compression: HiP-CT 3D organ image dataset.
  • Image Classification: CIFAR10, CIFAR100, Tiny-imagenet, and Imagenet.
  • Trajectory Fitting & Lane-Keeping: Used with vanilla Neural ODEs and Liquid Time Constant (LTC) architectures.
  • Autonomous Driving: DeepPiCar dataset.

Key Insights and Ideas

  • Learned Optimizers Aren't Always Best: While learned optimizers are designed to be general, they can sometimes over-explore or make suboptimal decisions, especially in fine-tuning. HUB shows that a hybrid approach can be more robust.
  • Gradient Magnitude as a Guide: Using the magnitude of gradients to dynamically switch between learned and hand-designed optimizers is an effective way to leverage their respective strengths. Small gradients benefit from exploration (learned), while large gradients benefit from stability (hand-designed).
  • Fine-tuning is Different: The paper highlights that fine-tuning requires a different strategy than training from scratch. Inverting the weighting in HUB for fine-tuning tasks proves crucial for preserving pre-trained knowledge.
  • Theoretical Justification: The authors provide a theoretical analysis showing how HUB can mitigate issues like gradient vanishing and exploding by intelligently combining optimizer behaviors.

Key Results

  • Dominant Performance: HUB outperforms all competitors in 94% of the tested tasks, achieving superior testing performance.
  • Robustness and Generalization: HUB demonstrates significant improvements in generalization across various model architectures (MLPs, CNNs, RNNs, Neural ODEs, Transformers) and tasks.
  • Fine-tuning Success: HUB effectively addresses VeLO's poor performance in fine-tuning tasks, a significant limitation of learned optimizers.
  • Specific Examples:
    • On image compression tasks (MLP), HUB achieved higher PSNR than Adamax and VeLO.
  • For Vision Transformer (ViT) image classification, HUB consistently achieved higher Top1/Top5 accuracy across CIFAR10, CIFAR100, and Tiny-imagenet.
  • In RNN and Neural ODE tasks (LSTM/LTC), HUB significantly outperformed baseline optimizers and VeLO, especially in challenging scenarios where others failed or encountered numerical issues (NaNs).

Limitations and Future Work

  • Computational Cost: While HUB aims to be efficient, the paper acknowledges that learned optimizers can have higher per-step overhead. Future work could focus on further optimizing this aspect.
  • Generalizability of HUB: The paper suggests that HUB's principles can be applied to other learned optimizers beyond VeLO, which would be a valuable area for future research.
  • Theoretical Exploration: While theoretical analysis is provided, further exploration into the theoretical underpinnings of HUB's stability and convergence properties could be beneficial.
  • Block-wise HUB Potential: The paper introduces block-wise HUB as a promising direction for handling complex network structures with pre-trained and newly added components, suggesting further investigation into optimal weighting strategies for different blocks.

Summary updated

Abstract

Learned optimizers are a crucial component of meta-learning. Recent advancements in scalable learned optimizers have demonstrated their superior performance over hand-designed optimizers in various tasks. However, certain characteristics of these models, such as an unstable learning curve, limited ability to handle unseen tasks and network architectures, difficult-to-control behaviours, and poor performance in fine-tuning tasks impede their widespread adoption. To tackle the issue of generalization in scalable learned optimizers, we propose a hybrid-update-based (HUB) optimization strategy inspired by recent advancements in hard prompt tuning and result selection techniques used in large language and vision models. This approach can be easily applied to any task that involves hand-designed or learned optimizer. By incorporating hand-designed optimizers as the second component in our hybrid approach, we are able to retain the benefits of learned optimizers while stabilizing the training process and, more importantly, improving testing performance. We validate our design through a total of 17 tasks, consisting of thirteen training from scratch and four fine-tuning settings. These tasks vary in model sizes, architectures, or dataset sizes, and the competing optimizers are hyperparameter-tuned. We outperform all competitors in 94% of the tasks with better testing performance. Furthermore, we conduct a theoretical analysis to examine the potential impact of our hybrid strategy on the behaviours and inherited traits of learned optimizers.
BibTeX
@misc{huang2023hub,
  author = {Tiejun Huang},
  title = {{HUB: Guiding Learned Optimizers with Continuous Prompt Tuning}},
  year = {2023},
  eprint = {2305.16823},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.16823},
}
Figure