TTE: Two Tokens Are Enough to Improve Parameter-Efficient Tuning

Jiacheng Ruan, Mingye Xie, Jingsheng Gao, Xian Gao, Suncheng Xiang, Ting Liu, Yuzhuo FuAAAI 2025 · DOI

Introduction

Fine-tuning large pre-trained models typically follows two paradigms: Full Parameter Tuning (FPT), which updates all model weights, and Parameter-Efficient Tuning (PET), which freezes the backbone and trains only a tiny fraction (often <1%) of the parameters. While PET is highly efficient, it faces a significant challenge: overfitting, particularly when downstream data is scarce (e.g., benchmarks with only 1,000 samples).

Standard regularization techniques like weight decay or Dropout often fail or even degrade performance in PET because the learnable parameter space is so constrained. TTE (Two Tokens are Enough) is a novel regularization framework designed to solve this persistent overfitting problem by introducing minimal learnable components and a specialized loss function.

The TTE Framework

The TTE framework introduces two main components that can be integrated into existing PET methods like VPT, Adapters, or LoRA:

1. Learnable Token Module

Instead of relying solely on the standard [CLS] token, TTE adds two new learnable vectors:

  • Global Token ($t^g$): A single vector learned across the entire downstream dataset to capture general task-specific knowledge.
  • Instance Token ($t^{in}$): A vector generated dynamically from each input image via a linear projection to capture specific image details.
  • Adaptive Dropout: A specialized layer with a learnable mask ratio that prevents these tokens from simply memorizing the training data.

2. Parameter-Free Cross Attention (PFCA) Loss

To regularize the model without adding heavy new weights, the authors utilize the final-layer outputs of the [CLS] token, $t^g$, and $t^{in}$. The PFCA loss forces the model to find a balance between general pre-trained features and specific downstream adaptations.

The PFCA loss architecture uses cross-attention between global, instance, and CLS tokens to regularize the model without adding parameters.

As shown in the diagram, the framework computes cross-attention twice: once using $t^g$ to query the [CLS] token (bridging global task info with pre-trained knowledge) and once using $t^g$ to query $t^{in}$ (bridging global info with instance info). The Mean Squared Error (MSE) between these two attention outputs ($O_1$ and $O_2$) serves as the regularization constraint.

Key Results and Insights

The framework was evaluated across the VTAB-1K benchmark (19 diverse tasks with 1,000 samples each) and Fine-Grained Visual Classification (FGVC) datasets.

Overcoming Overfitting

Training curves reveal that standard PET methods (Vanilla) and even advanced FPT regularization (DR-Tune) often suffer from a widening gap between training loss and validation accuracy. TTE stabilizes these curves, ensuring that the model continues to generalize well even as training progresses.

Comparison of training dynamics across different tuning methods. TTE (bottom row) shows significantly more stable accuracy and better convergence compared to Vanilla and DR-Tune.

Performance Highlights

  • Superior Accuracy: TTE improved the average performance of classic PET methods on VTAB-1K by 3.36%. A TTE-enhanced Adapter achieved 75.24% mean accuracy, surpassing DR-Tune (the leading FPT method) by 1%.
  • Extreme Efficiency: While DR-Tune requires tuning 85.84M parameters, TTE achieves better results with only 0.15M parameters—a 572x reduction in trainable overhead.
  • The "Two Token" Sweet Spot: The authors discovered that increasing the number of global or instance tokens beyond one actually decreased performance. In low-data regimes, "Two Tokens are Enough" to capture necessary information without introducing excessive noise.

Conclusion

TTE demonstrates that effective regularization for Parameter-Efficient Tuning requires a different approach than Full-Parameter Tuning. By using the pre-trained [CLS] token as an anchor and introducing just two specialized tokens, TTE provides a lightweight yet powerful way to prevent overfitting, allowing PET methods to finally outperform their full-parameter counterparts in data-scarce environments.

Summary updated

BibTeX
@inproceedings{ruan2025tte,
  author = {Jiacheng Ruan and Mingye Xie and Jingsheng Gao and Xian Gao and Suncheng Xiang and Ting Liu and Yuzhuo Fu},
  title = {{TTE: Two Tokens Are Enough to Improve Parameter-Efficient Tuning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i19.34226},
}
Figure