Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

TL;DR

Progtuning is a new framework that makes fine-tuning large language models more efficient by progressively "shrinking" the number of active Transformer blocks as training continues. By focusing updates on the most impactful high-level layers and freezing lower ones over time, it reduces the total number of parameter updates by approximately 25% while maintaining—and sometimes even improving—model accuracy.

What Problem Does This Paper Address?

As Transformer-based models like GPT-3 and LLaMA grow to hundreds of billions of parameters, traditional fine-tuning (updating every weight) has become prohibitively expensive. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA or Adapters help by targeting only a small subset of weights, they still typically update that same subset throughout the entire training process.

The authors identify a major inefficiency: not all Transformer blocks contribute equally to a downstream task. Their preliminary experiments showed that higher-level blocks (those closer to the output) have a much larger impact on performance than lower-level blocks (those closer to the input). Updating every block in every epoch is a waste of computational resources. Progtuning aims to optimize this resource allocation by dynamically adjusting which layers are "trainable" during the training run.

Methods

The core innovation is a strategy called Progressive Shrinking.

1. Partitioning: The model’s $L$ Transformer blocks are divided into $T$ parts (where $T$ is the number of training epochs).
2. Sequential Freezing:
   • In the first epoch, the entire model is trainable.
3. In the second epoch, the first "part" (the lowest layers) is frozen, and only the remaining layers are updated.
4. This continues until the final epoch, where only the highest-level blocks and the classification head are updated.
5. Compatibility: Progtuning is a "wrapper" framework. It can be applied to standard full fine-tuning or integrated with PEFT methods like Adapters, BitFit, and LoRA. When used with PEFT, Progtuning progressively freezes the adapter modules or low-rank matrices associated with the lower blocks.

Datasets

The researchers evaluated Progtuning across a wide variety of standard NLP benchmarks:

• GLUE (General Language Understanding Evaluation): A collection of nine tasks including sentiment analysis (SST-2), linguistic acceptability (CoLA), and natural language inference (MNLI, RTE).
• SQuAD (Stanford Question Answering Dataset) v1.1 and v2.0: Challenging reading comprehension tasks where the model must find answer spans in a text.

Key Insights and Ideas

• High Layers Matter More: The authors confirmed that for downstream tasks, the "semantic" information in higher layers is more critical for adaptation than the "low-level" features captured by early layers.
• Regularization Effect: Surprisingly, Progtuning often outperformed standard fine-tuning. The authors suggest that progressively freezing layers acts as a form of regularization, preventing the model from overfitting to the training data by limiting its capacity to change as the learning rate decays.
• Directionality is Key: They tested "Progressive Growing" (starting with few layers and adding more) and found it performed significantly worse. This is because lower layers need to be adjusted early when the learning rate is high; trying to train them in later epochs when the learning rate has decayed prevents them from acquiring sufficient knowledge.

Key Results

• Parameter Reduction: Progtuning reduced the total number of updated parameters by roughly 25% in standard fine-tuning scenarios.
• Performance Gains: On the GLUE benchmark, Progtuning actually improved the average score of RoBERTa-base by 0.7 points compared to full fine-tuning.
• PEFT Synergy: When combined with Adapter tuning, Progtuning reduced updated parameters by a staggering 67% (from 35.8M down to 11.9M) while staying within 0.5 points of the original Adapter performance.
• SQuAD Success: On SQuAD v1.1, the framework achieved a higher F1 score (88.1 vs 87.2) while updating 26% fewer parameters.

Limitations and Future Work

• Model Stability: The authors noted that $BERT_{LARGE}$ can be unstable on certain tasks (a known issue in the field), leading them to focus primarily on $BERT_{BASE}$ for their ablation studies.
• Training Time: While the paper notes that Progtuning reduces training time due to fewer backward pass operations, they did not include specific hardware-clocked time comparisons in the final results, noting that training time doesn't always perfectly reflect raw compute consumption.
• Heuristic Schedules: The current shrinking schedule is tied directly to the number of epochs. Future research could explore more sophisticated, data-driven schedules for freezing layers based on real-time gradient information.

@misc{ji2025progtuning,
  author = {Xihan Ji and Zhendong Zhao and Xiaojun Chen and Xin Zhao and Zhihang Liu},
  title = {{Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models}},
  year = {2025},
  eprint = {2506.21119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.21119},
  howpublished = {arXiv},
}