Introduction
Training state-of-the-art Vision Transformers (ViTs) is computationally intensive, often requiring massive datasets and weeks of compute. While initializing large models from smaller pretrained ones ("model scaling") is a common strategy, existing methods frequently fail to preserve the learned functionality effectively. These failures typically stem from ignoring optimizer states, creating "cold start" discrepancies between new and reused parameters, and attempting to scale multiple dimensions (width, depth, heads) simultaneously, which disrupts model stability.
The TripLe Framework
To address these challenges, we introduce TripLe, a framework built on three core principles: reusing pretrained models, progressive learning, and knowledge distillation. TripLe optimizes the scaling process through a serialized approach and the preservation of training momentum.
Three-Stage Training Pipeline
TripLe employs a structured pipeline to transition from a small pretrained model to a larger target architecture:
- Stage I (Width Scaling): The model's width is expanded using a functionality-preserving operator. A brief warmup period establishes training states for newly added parameters.
- Stage II (Depth Growing): Model depth is doubled by duplicating existing layers and their associated optimizer states. To prevent "pretrained knowledge wash-out," the bottom layers are frozen, allowing the new top layers to adapt.
- Stage III (Full Training): All layers are unfrozen for joint optimization to reach peak performance.
Optimizer State Transfer
A key innovation in TripLe is the scaling of AdamW optimizer states ($m$ and $v$) alongside model weights. By preserving the "momentum" built up during the small model's training, TripLe maintains the optimization trajectory, preventing the large model from having to "re-learn" the gradient direction.
Experimental Results
We evaluated TripLe primarily on ImageNet-1k using the DeiT architecture family, scaling from "Tiny" (5M parameters) to "Base" (86M parameters).
Training Efficiency and Accuracy
TripLe achieves massive speedups, saving between 71.0% and 80.9% of training time compared to training from scratch. Notably, a 44MB ViT trained via TripLe outperformed the official 86MB DeiT-B by 0.2%, demonstrating that "recycled" models can exceed the performance of larger models trained conventionally.
Neural Architecture Search (NAS)
TripLe serves as a high-fidelity proxy for NAS. By starting from a warm pretrained state, it increases the Kendall-tau correlation for architecture ranking from 0.22 (scratch) to 0.86. This allowed the discovery of a 27MB model that outperforms DeiT-B while using 69% fewer parameters and FLOPs.
Key Insights
- Momentum Preservation: Preserving gradient history is as critical as weight initialization for maintaining training efficiency.
- Mitigating the Cold Start: Freezing "warm" layers during the introduction of "cold" parameters prevents noisy early gradients from damaging pretrained knowledge.
- Orthogonality: TripLe is complementary to Knowledge Distillation (KD), providing an additional 1.8% accuracy boost when combined.
Conclusion
TripLe demonstrates that efficient ViT scaling is possible by carefully managing the transition of weights and optimizer states. While currently focused on the DeiT family, the principles of serialized scaling and momentum preservation offer a promising path for accelerating the development of diverse large-scale vision models.
Summary updated