Introduction
Fine-tuning large pre-trained models typically follows two paradigms: Full Parameter Tuning (FPT), which updates all model weights, and Parameter-Efficient Tuning (PET), which freezes the backbone and trains only a tiny fraction (often <1%) of the parameters. While PET is highly efficient, it faces a significant challenge: overfitting, particularly when downstream data is scarce (e.g., benchmarks with only 1,000 samples).
Standard regularization techniques like weight decay or Dropout often fail or even degrade performance in PET because the learnable parameter space is so constrained. TTE (Two Tokens are Enough) is a novel regularization framework designed to solve this persistent overfitting problem by introducing minimal learnable components and a specialized loss function.
The TTE Framework
The TTE framework introduces two main components that can be integrated into existing PET methods like VPT, Adapters, or LoRA:
1. Learnable Token Module
Instead of relying solely on the standard [CLS] token, TTE adds two new learnable vectors:
- Global Token ($t^g$): A single vector learned across the entire downstream dataset to capture general task-specific knowledge.
- Instance Token ($t^{in}$): A vector generated dynamically from each input image via a linear projection to capture specific image details.
- Adaptive Dropout: A specialized layer with a learnable mask ratio that prevents these tokens from simply memorizing the training data.
2. Parameter-Free Cross Attention (PFCA) Loss
To regularize the model without adding heavy new weights, the authors utilize the final-layer outputs of the [CLS] token, $t^g$, and $t^{in}$. The PFCA loss forces the model to find a balance between general pre-trained features and specific downstream adaptations.
As shown in the diagram, the framework computes cross-attention twice: once using $t^g$ to query the [CLS] token (bridging global task info with pre-trained knowledge) and once using $t^g$ to query $t^{in}$ (bridging global info with instance info). The Mean Squared Error (MSE) between these two attention outputs ($O_1$ and $O_2$) serves as the regularization constraint.
Key Results and Insights
The framework was evaluated across the VTAB-1K benchmark (19 diverse tasks with 1,000 samples each) and Fine-Grained Visual Classification (FGVC) datasets.
Overcoming Overfitting
Training curves reveal that standard PET methods (Vanilla) and even advanced FPT regularization (DR-Tune) often suffer from a widening gap between training loss and validation accuracy. TTE stabilizes these curves, ensuring that the model continues to generalize well even as training progresses.
Performance Highlights
- Superior Accuracy: TTE improved the average performance of classic PET methods on VTAB-1K by 3.36%. A TTE-enhanced Adapter achieved 75.24% mean accuracy, surpassing DR-Tune (the leading FPT method) by 1%.
- Extreme Efficiency: While DR-Tune requires tuning 85.84M parameters, TTE achieves better results with only 0.15M parameters—a 572x reduction in trainable overhead.
- The "Two Token" Sweet Spot: The authors discovered that increasing the number of global or instance tokens beyond one actually decreased performance. In low-data regimes, "Two Tokens are Enough" to capture necessary information without introducing excessive noise.
Conclusion
TTE demonstrates that effective regularization for Parameter-Efficient Tuning requires a different approach than Full-Parameter Tuning. By using the pre-trained [CLS] token as an anchor and introducing just two specialized tokens, TTE provides a lightweight yet powerful way to prevent overfitting, allowing PET methods to finally outperform their full-parameter counterparts in data-scarce environments.
Summary updated