Abstract
TLEG (Transformer as Linear Expansion of Learngene) introduces a novel initialization strategy for Transformers of arbitrary depth using just two shared parameter modules. By identifying an approximately linear trend in layer weights of trained Transformers, TLEG "expands" a condensed "learngene" to initialize various model sizes. This approach reduces training costs by 2x and storage requirements for initialization by up to 19x while maintaining or improving performance across multiple vision benchmarks.
Introduction
In the current era of foundation models, the standard approach involves pre-training massive architectures and fine-tuning them for specific tasks. However, this paradigm faces significant challenges in resource flexibility and efficiency. Deploying models across diverse hardware—from edge devices to cloud servers—usually requires pre-training and storing multiple models of different sizes separately.
TLEG builds on the concept of a learngene—a condensed unit of knowledge extracted from an "ancestry" model that can be used to initialize "descendant" models. Unlike previous attempts that were often inconsistent for Transformer architectures, TLEG leverages the discovery that layer weights follow a remarkably linear trend relative to their position in the stack.
Methodology: Linear Expansion of Learngenes
The core innovation of TLEG is the Linear Expansion strategy. By performing Principal Component Analysis (PCA) on well-trained Vision Transformers (ViTs), researchers found that weights do not change randomly but follow a predictable trajectory across the depth of the network.
The TLEG Pipeline
The TLEG framework operates in two primary stages:
- Learngene Learning (Stage 1): Instead of training a full model, the authors define two shared parameter modules, $\theta_A$ and $\theta_B$. Every layer $l$ in an auxiliary model (Aux-Net) is constrained by a simple linear formula: $$\theta_l = \theta_B + \frac{l-1}{L} \times \theta_A$$ This Aux-Net is trained by distilling knowledge from a large, pre-trained "Ancestry" model. Crucially, only the learngene parameters ($\theta_A$ and $\theta_B$) are updated during this stage.
- Initialization and Fine-tuning (Stage 2): To create a descendant model of any depth (e.g., 6-layer or 12-layer), the learned $\theta_A$ and $\theta_B$ are used to calculate the starting weights for each layer using the same linear formula. Once initialized, the linear constraint is removed, and the model is fine-tuned normally on the target dataset.
Key Results and Insights
The authors evaluated TLEG across a wide range of computer vision benchmarks, including ImageNet-1K and downstream tasks like CIFAR-100 and iNaturalist 2019.
- Performance and Efficiency: TLEG-initialized models consistently outperformed models trained from scratch while cutting training costs in half (~2x reduction). On CIFAR-100, TLEG outperformed standard initialization methods by a massive +7.66%.
- Storage Savings: To initialize a suite of models with different depths, TLEG requires 19x fewer parameters to be stored compared to traditional pre-training approaches.
- Elastic Model Production: Because the learngene consists of only two layers' worth of parameters, it allows for the generation of right-sized models for specific hardware constraints on the fly.
Conclusion and Future Work
TLEG demonstrates that Transformer layers exist on a continuum, and unique weights for every layer are not required at the start. By capturing the "direction" of weight evolution, TLEG provides a superior starting point for training. While currently focused on Vision Transformers and image classification, future work may explore non-linear expansion functions or apply the framework to other architectures like ConvNets or state-space models.
Summary updated