TL;DR
This paper introduces LoGAH, a novel Graph HyperNetwork (GHN) that efficiently predicts initial parameters for large Transformer models like ViT and GPT-2. By employing a low-rank decoder, LoGAH significantly reduces the parameter count and computational cost compared to previous GHNs, enabling better initializations for massive models that were previously out of reach for many researchers.
What Problem Does This Paper Address?
Pretraining large deep learning models from scratch is incredibly expensive and time-consuming, making it inaccessible for many researchers. While Graph HyperNetworks (GHNs) offer a promising solution by predicting model parameters, existing GHN approaches struggle with very wide networks. They often resort to parameter copying and require a disproportionately large number of parameters themselves, hindering their practical application for models with hundreds of millions or billions of parameters. This paper aims to bridge this gap by proposing a more efficient and scalable GHN for predicting parameters of large Transformer architectures.
Methods
The core innovation is LoGAH (Low-rank GrAph Hypernetworks). LoGAH builds upon the Graph HyperNetwork framework but introduces a low-rank parameter decoder. Traditional GHNs use a decoder whose parameter count scales cubically with its internal dimension ($O(d^3)$). LoGAH's decoder, however, uses a low-rank decomposition technique inspired by convolutional weight factorization. This reduces the decoder's parameter complexity to $O(d^2)$, allowing it to predict parameters for significantly wider networks without a proportional increase in its own size. Specifically, LoGAH can predict parameters for a 774-million parameter GPT-2 model using a LoGAH-TINY variant with only 2.5 million parameters.
Datasets
- VITS-1K: A newly created dataset of 1,000 diverse computational graphs for Vision Transformer (ViT) architectures. These graphs vary in the number of layers, heads, and hidden sizes, with parameter counts up to 10 million, designed to train GHNs for predicting parameters of larger ViT models.
- GPTS-1K: A similar dataset for GPT-2 architectures, also containing 1,000 computational graphs with varying architectural parameters, designed to train GHNs for predicting parameters of larger GPT-2 models. The maximum parameter count in these architectures is capped at 30 million.
- Standard Benchmarks: The paper also utilizes standard datasets for fine-tuning and evaluation, including CIFAR-10, CIFAR-100, and ImageNet for ViT experiments, and WikiText-2 and WikiText-103 for GPT-2 experiments.
Key Insights and Ideas
- Low-rank factorization is key to scalability: The paper demonstrates that applying low-rank decomposition to the decoder's parameter prediction mechanism is crucial for handling the vast parameter spaces of modern large models without ballooning the hypernetwork's size.
- Efficiency without sacrificing performance: LoGAH achieves better performance than random initialization and existing GHNs while using significantly fewer parameters, highlighting a more efficient path to good initializations.
- Transfer learning capability: LoGAH exhibits promising transfer learning abilities, where models trained on smaller datasets can effectively predict parameters for larger, more complex tasks, reducing the need for massive training datasets for the hypernetwork itself.
- Generalization across modalities: LoGAH shows effectiveness for both vision (ViT) and language (GPT-2) Transformer models, indicating its broad applicability.
Key Results
- LoGAH initialized ViT-Small and ViT-Base models achieved superior accuracy on CIFAR-10, CIFAR-100, and ImageNet compared to random initialization, orthogonal initialization, and GHN-3. For instance, on CIFAR-100, LoGAH-LARGE outperformed random initialization by 3.46% on ViT-Small and 6.47% on ViT-Base.
- LoGAH-TINY, with only 2.5M parameters, could predict parameters for GPT-2-Large (774M parameters), a feat that existing methods struggled with.
- In GPT-2 experiments on WikiText-103, LoGAH significantly improved perplexity scores for GPT-2-Medium and GPT-2-Large compared to random initialization. For GPT-2-Large, LoGAH-T/m2 achieved a perplexity of 27.18, compared to 32.41 for random initialization.
- The paper shows that LoGAH can be trained on smaller datasets (like CIFAR-10) and then used to initialize models for larger, more challenging tasks (like CIFAR-100 or ImageNet), demonstrating effective transfer learning.
Limitations and Future Work
- GPT-2 experiments limited: Due to time and resource constraints, GPT-2 experiments were primarily conducted on the WikiText dataset and only with the two smallest LoGAH variants (TINY and SMALL).
- Novel architectures: The paper acknowledges that for drastically novel architectures, the GHN might need to be trained on data with similar distributions to avoid performance degradation.
- Future work: The authors suggest exploring LoGAH's ability on even larger, modern LLMs and investigating its performance on architectures significantly different from those it was trained on. They also highlight the potential for reducing LOGAH's training time by leveraging its transfer learning capabilities.
Summary updated