Introduction
Vision Transformers (ViTs) have fundamentally changed the landscape of computer vision, yet they are often treated as "black boxes." While architectural scaling (adding layers or changing patch sizes) is the standard approach for performance gains, it often involves significant trial and error and high computational costs. This paper introduces ViT-Calibrator, a two-stage optimization paradigm that improves ViT performance by "calibrating" the internal decision stream—the flow of information from input patches to the final classification token.
Identifying Inefficiencies in the Decision Stream
The researchers identify two primary bottlenecks in standard ViT architectures:
- Information Dilution: Shallow layers frequently struggle to distinguish foreground objects from background noise, leading to inefficient propagation of semantic information.
- Dimensional Noise: The class token ($[CLS]$), used for final predictions, contains hundreds of dimensions. The authors discovered that for any specific category, a large portion of these dimensions are irrelevant "noise" that can destabilize accuracy.
Methodology: Two-Stage Calibration
ViT-Calibrator addresses these inefficiencies through a biologically-inspired feedback loop and a dimension-filtering mechanism.
1. Token-level Calibration (The Feedback Loop)
Inspired by biological neural feedback, the authors designed a mechanism where high-level semantic information from deep layers is fed back into shallow layers. By projecting features from deep and shallow layers into a shared semantic space, the model generates a "feedback offset." This offset adjusts the self-attention mechanism in early layers, effectively guiding the model to prioritize regions that deeper layers have already identified as semantically important.
2. Dimension-level Calibration (The Filter)
Using Layer-wise Relevance Propagation (LRP), the authors analyzed which specific dimensions of the class token contribute to correct predictions. They found that relevance is sparse and category-specific. To exploit this, they created a "category-aware mask" that zeros out noisy dimensions. During fine-tuning, a teacher-student distillation setup is used: a "teacher" model (using the mask) guides the "student" (the original model) to focus only on the dimensions that truly matter for the target category.
Experimental Results
The authors evaluated ViT-Calibrator across seven architectures, including ViT, DeiT, and Eva-L, on benchmarks such as CIFAR-100, ImageNet-1K, and ImageNet-Real.
Key findings include:
- Consistent Accuracy Gains: On ImageNet-1K, DeiT-T saw a significant jump from 72.17% to 74.19% (+2.02%). Even massive models like Eva-L showed measurable improvements.
- Improved Focus: Attention heatmaps revealed that calibrated models exhibit a much sharper focus on foreground objects in intermediate layers compared to baseline models.
- Efficiency: Unlike training from scratch, ViT-Calibrator works as a fine-tuning step on pre-trained models, making it a practical tool for researchers with limited compute resources.
Conclusion
ViT-Calibrator offers a novel way to enhance Vision Transformers by refining how they process and select information. By accelerating the natural "pruning" of background noise and filtering out dimensional interference in the class token, the method provides a robust 1–2% accuracy boost across diverse datasets and architectures. Future work may focus on making the dimensional masks more dynamic to further optimize performance on edge devices.
Summary updated