Introduction: The Inductive Bias Gap
Convolutional Neural Networks (CNNs) have long dominated computer vision due to their inherent "inductive bias"—architectural assumptions like locality and translation invariance. These biases allow CNNs to learn effectively even from relatively small datasets. In contrast, Visual Transformers (VTs) lack these built-in spatial priors. While this allows VTs to capture complex global relationships, it also makes them notoriously data-hungry. Without massive pre-training on datasets like ImageNet-21K or JFT-300M, VTs often underperform CNNs on the "medium-small" datasets typically encountered in specialized real-world applications.
Methodology: Dense Relative Localization
To bridge this gap, this research introduces Dense Relative Localization ($L_{drloc}$), a simple, architecture-agnostic auxiliary task designed to instill "spatial common sense" into VTs during training.
The core mechanism involves forcing the model to explicitly reason about the geometric relationships between different parts of an image. As shown in the architectural overview below, the process follows three main steps:
- Token Sampling: From the final grid of token embeddings produced by the Transformer, multiple pairs of tokens are randomly selected.
- Distance Prediction: These pairs are fed into a lightweight, two-layer Multi-Layer Perceptron (MLP) that predicts their normalized 2D geometric distance ($d_u, d_v$).
- Joint Optimization: The model is trained using a multi-task loss function that combines standard cross-entropy for classification with the $L_{drloc}$ loss for spatial reasoning.
By tasking the model with predicting "how far is Patch A from Patch B?", the VT is incentivized to encode both local features and global context into every token, effectively compensating for the lack of convolutional bias.
Experimental Setup and Datasets
The effectiveness of $L_{drloc}$ was evaluated across 11 diverse datasets, ranging from standard benchmarks like CIFAR-10/100 and SVHN to fine-grained classification (Oxford Flowers102) and complex domain adaptation tasks (DomainNet). These datasets represent a wide spectrum of visual styles and data scales, providing a rigorous test for the method's robustness.
Key Results and Performance Gains
The inclusion of the $L_{drloc}$ task yields significant improvements in classification accuracy, particularly when training data is scarce.
One of the most striking results was observed on the "Quickdraw" dataset using a Swin-T architecture, where Top-1 accuracy surged from 24.08% to 69.41%—a gain of over 45 percentage points. Beyond raw accuracy, the auxiliary task also accelerates convergence; models trained with $L_{drloc}$ for 60 epochs often outperformed baselines trained for 100 epochs.
As illustrated in the tables above, the method is effective across various "second-generation" VTs like CvT and T2T-ViT. While these hybrid models already incorporate some convolutional layers and are thus more robust than pure ViTs, they still benefit significantly from the explicit spatial supervision provided by $L_{drloc}$.
Insights and Limitations
The research highlights that VTs do not necessarily require millions of images if provided with a "spatial nudge." The $L_{drloc}$ loss acts as a powerful regularizer, filling the gap left by the absence of inductive bias. Furthermore, the method is highly sample-efficient, adding only about 2% to the total training time.
However, some limitations remain. The task's effectiveness is sensitive to the grid resolution; while a 7x7 grid works well, a 14x14 grid proved too challenging, leading to noisier gradients. Additionally, the optimal weight for the auxiliary loss ($\lambda$) varies slightly by architecture, requiring some hyperparameter tuning for peak performance. Future work may explore scaling this approach to higher-resolution feature maps and non-grid-based vision tasks.
Summary updated