TripLe: Revisiting Pretrained Model Reuse and Progressive Learning for Efficient Vision Transformer Scaling and Searching

Cheng Fu, Hanxian Huang, Zixuan Jiang, Yun Ni, Lifeng Nai, Gang Wu, Liqun Cheng, Yanqi Zhou, Sheng Li, Andrew Li, Jishen Zhao

Introduction

Training state-of-the-art Vision Transformers (ViTs) is computationally intensive, often requiring massive datasets and weeks of compute. While initializing large models from smaller pretrained ones ("model scaling") is a common strategy, existing methods frequently fail to preserve the learned functionality effectively. These failures typically stem from ignoring optimizer states, creating "cold start" discrepancies between new and reused parameters, and attempting to scale multiple dimensions (width, depth, heads) simultaneously, which disrupts model stability.

The TripLe Framework

To address these challenges, we introduce TripLe, a framework built on three core principles: reusing pretrained models, progressive learning, and knowledge distillation. TripLe optimizes the scaling process through a serialized approach and the preservation of training momentum.

Three-Stage Training Pipeline

TripLe employs a structured pipeline to transition from a small pretrained model to a larger target architecture:

  1. Stage I (Width Scaling): The model's width is expanded using a functionality-preserving operator. A brief warmup period establishes training states for newly added parameters.
  2. Stage II (Depth Growing): Model depth is doubled by duplicating existing layers and their associated optimizer states. To prevent "pretrained knowledge wash-out," the bottom layers are frozen, allowing the new top layers to adapt.
  3. Stage III (Full Training): All layers are unfrozen for joint optimization to reach peak performance.
The TripLe three-stage training pipeline illustrating width scaling and depth growing.

Optimizer State Transfer

A key innovation in TripLe is the scaling of AdamW optimizer states ($m$ and $v$) alongside model weights. By preserving the "momentum" built up during the small model's training, TripLe maintains the optimization trajectory, preventing the large model from having to "re-learn" the gradient direction.

Experimental Results

We evaluated TripLe primarily on ImageNet-1k using the DeiT architecture family, scaling from "Tiny" (5M parameters) to "Base" (86M parameters).

Training Efficiency and Accuracy

TripLe achieves massive speedups, saving between 71.0% and 80.9% of training time compared to training from scratch. Notably, a 44MB ViT trained via TripLe outperformed the official 86MB DeiT-B by 0.2%, demonstrating that "recycled" models can exceed the performance of larger models trained conventionally.

Training efficiency comparison: TripLe variants significantly outperform training from scratch in convergence speed and final accuracy.

Neural Architecture Search (NAS)

TripLe serves as a high-fidelity proxy for NAS. By starting from a warm pretrained state, it increases the Kendall-tau correlation for architecture ranking from 0.22 (scratch) to 0.86. This allowed the discovery of a 27MB model that outperforms DeiT-B while using 69% fewer parameters and FLOPs.

Key Insights

  • Momentum Preservation: Preserving gradient history is as critical as weight initialization for maintaining training efficiency.
  • Mitigating the Cold Start: Freezing "warm" layers during the introduction of "cold" parameters prevents noisy early gradients from damaging pretrained knowledge.
  • Orthogonality: TripLe is complementary to Knowledge Distillation (KD), providing an additional 1.8% accuracy boost when combined.

Conclusion

TripLe demonstrates that efficient ViT scaling is possible by carefully managing the transition of weights and optimizer states. While currently focused on the DeiT family, the principles of serialized scaling and momentum preservation offer a promising path for accelerating the development of diverse large-scale vision models.

Summary updated

Abstract

One promising way to accelerate transformer training is to reuse small pretrained models to initialize the transformer, as their existing representation power facilitates faster model convergence. Previous works designed expansion operators to scale up pretrained models to the target model before training. Yet, model functionality is difficult to preserve when scaling a transformer in all dimensions at once. Moreover, maintaining the pretrained optimizer states for weights is critical for model scaling, whereas the new weights added during expansion lack these states in pre-trained models. To address these issues, we propose TripLe, which partially scales a model before training, while growing the rest of the new parameters during training by copying both the warmed-up weights with the optimizer states from existing weights. As such, the new parameters introduced during training will obtain their training states. Furthermore, through serializing the scaling of model width and depth, the functionality of each expansion can be preserved. We evaluate TripLe in both single-trial model scaling and multi-trial neural architecture search (NAS). Due to the fast training convergence of TripLe, the proxy accuracy from TripLe better reveals the model quality compared to from-scratch training in multi-trial NAS. Experiments show that TripLe outperforms from-scratch training and knowledge distillation (KD) in both training time and task performance. TripLe can also be combined with KD to achieve an even higher task accuracy. For NAS, the model obtained from TripLe outperforms DeiT-B in task accuracy with 69% reduction in parameter size and FLOPs.
BibTeX
@inproceedings{fu2023triple,
  author = {Cheng Fu and Hanxian Huang and Zixuan Jiang and Yun Ni and Lifeng Nai and Gang Wu and Liqun Cheng and Yanqi Zhou and Sheng Li and Andrew Li and Jishen Zhao},
  title = {{TripLe: Revisiting Pretrained Model Reuse and Progressive Learning for Efficient Vision Transformer Scaling and Searching}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01573},
}
Figure