Transformer as Linear Expansion of Learngene

Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, Xin Geng

Abstract

TLEG (Transformer as Linear Expansion of Learngene) introduces a novel initialization strategy for Transformers of arbitrary depth using just two shared parameter modules. By identifying an approximately linear trend in layer weights of trained Transformers, TLEG "expands" a condensed "learngene" to initialize various model sizes. This approach reduces training costs by 2x and storage requirements for initialization by up to 19x while maintaining or improving performance across multiple vision benchmarks.

Introduction

In the current era of foundation models, the standard approach involves pre-training massive architectures and fine-tuning them for specific tasks. However, this paradigm faces significant challenges in resource flexibility and efficiency. Deploying models across diverse hardware—from edge devices to cloud servers—usually requires pre-training and storing multiple models of different sizes separately.

Comparison of TLEG with traditional learning paradigms.

TLEG builds on the concept of a learngene—a condensed unit of knowledge extracted from an "ancestry" model that can be used to initialize "descendant" models. Unlike previous attempts that were often inconsistent for Transformer architectures, TLEG leverages the discovery that layer weights follow a remarkably linear trend relative to their position in the stack.

Methodology: Linear Expansion of Learngenes

The core innovation of TLEG is the Linear Expansion strategy. By performing Principal Component Analysis (PCA) on well-trained Vision Transformers (ViTs), researchers found that weights do not change randomly but follow a predictable trajectory across the depth of the network.

The TLEG Pipeline

The TLEG framework operates in two primary stages:

  1. Learngene Learning (Stage 1): Instead of training a full model, the authors define two shared parameter modules, $\theta_A$ and $\theta_B$. Every layer $l$ in an auxiliary model (Aux-Net) is constrained by a simple linear formula: $$\theta_l = \theta_B + \frac{l-1}{L} \times \theta_A$$ This Aux-Net is trained by distilling knowledge from a large, pre-trained "Ancestry" model. Crucially, only the learngene parameters ($\theta_A$ and $\theta_B$) are updated during this stage.
Stage 1 of the TLEG pipeline: Learngene learning through distillation and linear mapping.
  1. Initialization and Fine-tuning (Stage 2): To create a descendant model of any depth (e.g., 6-layer or 12-layer), the learned $\theta_A$ and $\theta_B$ are used to calculate the starting weights for each layer using the same linear formula. Once initialized, the linear constraint is removed, and the model is fine-tuned normally on the target dataset.

Key Results and Insights

The authors evaluated TLEG across a wide range of computer vision benchmarks, including ImageNet-1K and downstream tasks like CIFAR-100 and iNaturalist 2019.

  • Performance and Efficiency: TLEG-initialized models consistently outperformed models trained from scratch while cutting training costs in half (~2x reduction). On CIFAR-100, TLEG outperformed standard initialization methods by a massive +7.66%.
  • Storage Savings: To initialize a suite of models with different depths, TLEG requires 19x fewer parameters to be stored compared to traditional pre-training approaches.
  • Elastic Model Production: Because the learngene consists of only two layers' worth of parameters, it allows for the generation of right-sized models for specific hardware constraints on the fly.

Conclusion and Future Work

TLEG demonstrates that Transformer layers exist on a continuum, and unique weights for every layer are not required at the start. By capturing the "direction" of weight evolution, TLEG provides a superior starting point for training. While currently focused on Vision Transformers and image classification, future work may explore non-linear expansion functions or apply the framework to other architectures like ConvNets or state-space models.

Summary updated

BibTeX
@inproceedings{xia2024transformer,
  author = {Shiyu Xia and Miaosen Zhang and Xu Yang and Ruiming Chen and Haokun Chen and Xin Geng},
  title = {{Transformer as Linear Expansion of Learngene}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i14.29533},
}
Figure