LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameters

Xinyu Zhou, Boris Knyazev, Alexia Jolicoeur-Martineau, Jie FuarXiv 2024 · OpenReview

TL;DR

This paper introduces LoGAH, a novel Graph HyperNetwork (GHN) that efficiently predicts initial parameters for large Transformer models like ViT and GPT-2. By employing a low-rank decoder, LoGAH significantly reduces the parameter count and computational cost compared to previous GHNs, enabling better initializations for massive models that were previously out of reach for many researchers.

What Problem Does This Paper Address?

Pretraining large deep learning models from scratch is incredibly expensive and time-consuming, making it inaccessible for many researchers. While Graph HyperNetworks (GHNs) offer a promising solution by predicting model parameters, existing GHN approaches struggle with very wide networks. They often resort to parameter copying and require a disproportionately large number of parameters themselves, hindering their practical application for models with hundreds of millions or billions of parameters. This paper aims to bridge this gap by proposing a more efficient and scalable GHN for predicting parameters of large Transformer architectures.

Methods

The core innovation is LoGAH (Low-rank GrAph Hypernetworks). LoGAH builds upon the Graph HyperNetwork framework but introduces a low-rank parameter decoder. Traditional GHNs use a decoder whose parameter count scales cubically with its internal dimension ($O(d^3)$). LoGAH's decoder, however, uses a low-rank decomposition technique inspired by convolutional weight factorization. This reduces the decoder's parameter complexity to $O(d^2)$, allowing it to predict parameters for significantly wider networks without a proportional increase in its own size. Specifically, LoGAH can predict parameters for a 774-million parameter GPT-2 model using a LoGAH-TINY variant with only 2.5 million parameters.

Datasets

  • VITS-1K: A newly created dataset of 1,000 diverse computational graphs for Vision Transformer (ViT) architectures. These graphs vary in the number of layers, heads, and hidden sizes, with parameter counts up to 10 million, designed to train GHNs for predicting parameters of larger ViT models.
  • GPTS-1K: A similar dataset for GPT-2 architectures, also containing 1,000 computational graphs with varying architectural parameters, designed to train GHNs for predicting parameters of larger GPT-2 models. The maximum parameter count in these architectures is capped at 30 million.
  • Standard Benchmarks: The paper also utilizes standard datasets for fine-tuning and evaluation, including CIFAR-10, CIFAR-100, and ImageNet for ViT experiments, and WikiText-2 and WikiText-103 for GPT-2 experiments.

Key Insights and Ideas

  • Low-rank factorization is key to scalability: The paper demonstrates that applying low-rank decomposition to the decoder's parameter prediction mechanism is crucial for handling the vast parameter spaces of modern large models without ballooning the hypernetwork's size.
  • Efficiency without sacrificing performance: LoGAH achieves better performance than random initialization and existing GHNs while using significantly fewer parameters, highlighting a more efficient path to good initializations.
  • Transfer learning capability: LoGAH exhibits promising transfer learning abilities, where models trained on smaller datasets can effectively predict parameters for larger, more complex tasks, reducing the need for massive training datasets for the hypernetwork itself.
  • Generalization across modalities: LoGAH shows effectiveness for both vision (ViT) and language (GPT-2) Transformer models, indicating its broad applicability.

Key Results

  • LoGAH initialized ViT-Small and ViT-Base models achieved superior accuracy on CIFAR-10, CIFAR-100, and ImageNet compared to random initialization, orthogonal initialization, and GHN-3. For instance, on CIFAR-100, LoGAH-LARGE outperformed random initialization by 3.46% on ViT-Small and 6.47% on ViT-Base.
  • LoGAH-TINY, with only 2.5M parameters, could predict parameters for GPT-2-Large (774M parameters), a feat that existing methods struggled with.
  • In GPT-2 experiments on WikiText-103, LoGAH significantly improved perplexity scores for GPT-2-Medium and GPT-2-Large compared to random initialization. For GPT-2-Large, LoGAH-T/m2 achieved a perplexity of 27.18, compared to 32.41 for random initialization.
  • The paper shows that LoGAH can be trained on smaller datasets (like CIFAR-10) and then used to initialize models for larger, more challenging tasks (like CIFAR-100 or ImageNet), demonstrating effective transfer learning.

Limitations and Future Work

  • GPT-2 experiments limited: Due to time and resource constraints, GPT-2 experiments were primarily conducted on the WikiText dataset and only with the two smallest LoGAH variants (TINY and SMALL).
  • Novel architectures: The paper acknowledges that for drastically novel architectures, the GHN might need to be trained on data with similar distributions to avoid performance degradation.
  • Future work: The authors suggest exploring LoGAH's ability on even larger, modern LLMs and investigating its performance on architectures significantly different from those it was trained on. They also highlight the potential for reducing LOGAH's training time by leveraging its transfer learning capabilities.

Summary updated

Abstract

A good initialization of deep learning models is essential since it can help them converge better and faster. However, pretraining large models is unaffordable for many researchers, which makes a desired prediction for initial parameters more necessary nowadays. Graph HyperNetworks (GHNs), one approach to predicting model parameters, have recently shown strong performance in initializing large vision models. Unfortunately, predicting parameters of very wide networks relies on copying small chunks of parameters multiple times and requires an extremely large number of parameters to support full prediction, which greatly hinders its adoption in practice. To address this limitation, we propose LoGAH (Low-rank GrAph Hypernetworks), a GHN with a low-rank parameter decoder that expands to significantly wider networks without requiring as excessive increase of parameters as in previous attempts. LoGAH allows us to predict the parameters of 774-million large neural networks in a memory-efficient manner. We show that vision and language models (i.e., ViT and GPT-2) initialized with LoGAH achieve better performance than those initialized randomly or using existing hypernetworks. Furthermore, we show promising transfer learning results w.r.t. training LoGAH on small datasets and using the predicted parameters to initialize for larger tasks. We provide the codes in https://github.com/Blackzxy/LoGAH .
BibTeX
@misc{zhou2024logah,
  author = {Xinyu Zhou and Boris Knyazev and Alexia Jolicoeur-Martineau and Jie Fu},
  title = {{LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameters}},
  year = {2024},
  eprint = {2405.16287},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.16287},
}

Loading related content...

Figure