Efficient Training of Visual Transformers with Small Datasets

Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, Marco De NadaiZenodo 2022 · DOI · OpenAlexGitHub resource

Introduction: The Inductive Bias Gap

Convolutional Neural Networks (CNNs) have long dominated computer vision due to their inherent "inductive bias"—architectural assumptions like locality and translation invariance. These biases allow CNNs to learn effectively even from relatively small datasets. In contrast, Visual Transformers (VTs) lack these built-in spatial priors. While this allows VTs to capture complex global relationships, it also makes them notoriously data-hungry. Without massive pre-training on datasets like ImageNet-21K or JFT-300M, VTs often underperform CNNs on the "medium-small" datasets typically encountered in specialized real-world applications.

Methodology: Dense Relative Localization

To bridge this gap, this research introduces Dense Relative Localization ($L_{drloc}$), a simple, architecture-agnostic auxiliary task designed to instill "spatial common sense" into VTs during training.

The core mechanism involves forcing the model to explicitly reason about the geometric relationships between different parts of an image. As shown in the architectural overview below, the process follows three main steps:

  1. Token Sampling: From the final grid of token embeddings produced by the Transformer, multiple pairs of tokens are randomly selected.
  2. Distance Prediction: These pairs are fed into a lightweight, two-layer Multi-Layer Perceptron (MLP) that predicts their normalized 2D geometric distance ($d_u, d_v$).
  3. Joint Optimization: The model is trained using a multi-task loss function that combines standard cross-entropy for classification with the $L_{drloc}$ loss for spatial reasoning.
Overview of the VT architecture with the proposed Localization MLP (f) auxiliary task.

By tasking the model with predicting "how far is Patch A from Patch B?", the VT is incentivized to encode both local features and global context into every token, effectively compensating for the lack of convolutional bias.

Experimental Setup and Datasets

The effectiveness of $L_{drloc}$ was evaluated across 11 diverse datasets, ranging from standard benchmarks like CIFAR-10/100 and SVHN to fine-grained classification (Oxford Flowers102) and complex domain adaptation tasks (DomainNet). These datasets represent a wide spectrum of visual styles and data scales, providing a rigorous test for the method's robustness.

Key Results and Performance Gains

The inclusion of the $L_{drloc}$ task yields significant improvements in classification accuracy, particularly when training data is scarce.

One of the most striking results was observed on the "Quickdraw" dataset using a Swin-T architecture, where Top-1 accuracy surged from 24.08% to 69.41%—a gain of over 45 percentage points. Beyond raw accuracy, the auxiliary task also accelerates convergence; models trained with $L_{drloc}$ for 60 epochs often outperformed baselines trained for 100 epochs.

Quantitative results showing Top-1 accuracy improvements across different VT architectures and sensitivity to the loss weight λ.

As illustrated in the tables above, the method is effective across various "second-generation" VTs like CvT and T2T-ViT. While these hybrid models already incorporate some convolutional layers and are thus more robust than pure ViTs, they still benefit significantly from the explicit spatial supervision provided by $L_{drloc}$.

Insights and Limitations

The research highlights that VTs do not necessarily require millions of images if provided with a "spatial nudge." The $L_{drloc}$ loss acts as a powerful regularizer, filling the gap left by the absence of inductive bias. Furthermore, the method is highly sample-efficient, adding only about 2% to the total training time.

However, some limitations remain. The task's effectiveness is sensitive to the grid resolution; while a 7x7 grid works well, a 14x14 grid proved too challenging, leading to noisier gradients. Additionally, the optimal weight for the auxiliary loss ($\lambda$) varies slightly by architecture, requiring some hyperparameter tuning for peak performance. Future work may explore scaling this approach to higher-resolution feature maps and non-grid-based vision tasks.

Summary updated

Abstract

Visual Transformers (VTs) are emerging as an architectural paradigm alternative to<br> Convolutional networks (CNNs). Differently from CNNs, VTs can capture global<br> relations between image elements and they potentially have a larger representation<br> capacity. However, the lack of the typical convolutional inductive bias makes these<br> models more data hungry than common CNNs. In fact, some local properties of the<br> visual domain which are embedded in the CNN architectural design, in VTs should<br> be learned from samples. In this paper, we empirically analyse different VTs,<br> comparing their robustness in a small training set regime, and we show that, despite<br> having a comparable accuracy when trained on ImageNet, their performance on<br> smaller datasets can be largely different. Moreover, we propose an auxiliary selfsupervised<br> task which can extract additional information from images with only a<br> negligible computational overhead. This task encourages the VTs to learn spatial<br> relations within an image and makes the VT training much more robust when<br> training data is scarce. Our task is used jointly with the standard (supervised)<br> training and it does not depend on specific architectural choices, thus it can be<br> easily plugged in the existing VTs. Using an extensive evaluation with different<br> VTs and datasets, we show that our method can improve (sometimes dramatically)<br> the final accuracy of the VTs. Our code is available at: https://github.com/<br> yhlleo/VTs-Drloc.
BibTeX
@inproceedings{liu2022efficient,
  author = {Yahui Liu and Enver Sangineto and Wei Bi and Nicu Sebe and Bruno Lepri and Marco De Nadai},
  title = {{Efficient Training of Visual Transformers with Small Datasets}},
  booktitle = {Zenodo},
  year = {2022},
  doi = {10.5281/zenodo.6363240},
}

Loading related content...

Figure