Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers

Bum Jun Kim, Sang Woo KimAAAI 2025 · DOI

TL;DR

Vision Transformers (ViTs) rely heavily on data augmentation for successful training, but common techniques like Mixup can introduce a "variance shift" in positional embeddings, degrading performance. This paper identifies this issue and proposes simple, effective rescaling strategies for positional embeddings during inference to mitigate this shift and improve ViT performance across various tasks.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have revolutionized computer vision, often outperforming traditional Convolutional Neural Networks (CNNs). However, ViTs are notoriously data-hungry. To compensate, researchers widely employ data augmentation techniques like Mixup, Cutmix, and random erasing. While these methods are crucial for training CNNs, this paper highlights a critical vulnerability when they are applied to ViTs: they can disrupt the positional embeddings, which are essential for ViTs to understand spatial relationships between image patches. Specifically, augmentations like Mixup can cause a "variance shift" in these embeddings. This means the way positional information is encoded during training differs from how it's handled during inference, leading to inconsistent behavior and reduced accuracy. The paper aims to diagnose this problem and provide practical solutions.

Methods

The core of the paper's contribution lies in its theoretical analysis and practical solutions for the identified variance shift.

  • Theoretical Analysis: The authors delve into the early stages of ViT architecture, particularly the interaction between patch embeddings and positional embeddings within the Layer Normalization (LN) layer. They mathematically demonstrate how the variance of the positional embedding relative to the patch embedding influences its contribution. Crucially, they extend this to the training-test phase, arguing that a consistent ratio of variances (Var[Image]/Var[Positional Embedding]) is needed. They then analyze common data augmentation techniques:
  • Upsampling: They show that interpolation-based upsampling (like bicubic) for positional embeddings during testing (to match varying input image sizes) introduces a variance shift (Var[UP(P)] ≠ Var[P]), while duplication-based upsampling (like nearest neighbor) conserves variance.
  • Mixup: They mathematically prove that Mixup inherently decreases variance (Var[Mixup Sample] < Var[Original Sample]), leading to a variance shift.
  • Cutmix: They demonstrate that Cutmix conserves variance, making it safe from this specific issue.
  • Random Erasing: They analyze Random Erasing and find that it conserves variance only when used in "pixel mode" and combined with specific mean-std normalization statistics (default-mean-std).
  • Proposed Solution: To counteract the variance shift caused by upsampling, the authors propose a simple rescaling of the positional embedding during the test phase: P_test = UP(P) / √k, where k is the variance ratio introduced by the upsampling method. This effectively calibrates the positional embedding's variance to match that of the training phase.

Datasets

  • ImageNet: Used for evaluating ViT performance on image classification tasks. The authors examined various spatial sizes of the test images (from 224x224 to 864x864).
  • ADE20K, LoveDA, Cityscapes: Used for evaluating ViT performance on semantic segmentation tasks.

Key Insights and Ideas

  • Positional Embedding Variance is Key: The paper convincingly shows that the variance of positional embeddings, not just their values, plays a critical role in ViT performance, especially when interacting with patch embeddings in early layers.
  • Augmentations Aren't Universally Good: Techniques that work wonders for CNNs can have unintended negative consequences for ViTs due to architectural differences, particularly concerning positional information.
  • Mixup's Downside: While Mixup is popular for boosting CNN performance, its inherent variance-reducing property makes it problematic for ViTs, potentially outweighing its benefits.
  • Simple Fixes Go a Long Way: The proposed 1/√k rescaling for positional embeddings is a remarkably simple yet effective solution to a complex problem, requiring no retraining of the ViT model.
  • Context Matters for Random Erasing: The effectiveness of Random Erasing is highly dependent on its configuration (pixel mode) and the normalization statistics used, highlighting the need for careful implementation.

Key Results

  • Image Classification: Rescaling positional embeddings with 1/√k consistently improved top-1 accuracy on ImageNet when using test images larger than the training resolution (e.g., 288x288 and above), recovering performance lost due to variance shift. For example, for ViT-L/16, accuracy improved by up to +4.18% at 864x864 resolution.
  • Semantic Segmentation: The proposed rescaling also led to consistent improvements across various metrics (aAcc, mIoU, mAcc) on ADE20K, LoveDA, and Cityscapes datasets, demonstrating its broad applicability. For instance, on LoveDA, aAcc improved by +0.88%.
  • Mixup vs. Cutmix: Experiments showed that Cutmix significantly improved ImageNet training performance, while adding Mixup (even with a low ratio) degraded performance, confirming the theoretical findings.
  • Random Erasing Configuration: The paper validated that using "pixel mode" random erasing with "default-mean-std" normalization yielded better results than other configurations on ImageNet.

Limitations and Future Work

  • Focus on Absolute Positional Embeddings: The study specifically targets ViTs with absolute positional embeddings. Future work could explore if similar variance shift issues and solutions exist for ViTs employing relative positional embeddings.
  • Mean Consistency: While the paper primarily focuses on variance consistency, it acknowledges that mean consistency also plays a role in Layer Normalization. A more comprehensive analysis could explore both aspects simultaneously.
  • Hyperparameter Tuning for k: Although the paper provides k values derived from empirical measurements (Table 2), these are presented as fixed values. Further research could investigate if these values need fine-tuning for specific architectures or datasets.
  • Broader Augmentation Landscape: The paper analyzes a select set of popular augmentations. Investigating other advanced augmentation techniques for their impact on ViT positional embeddings would be valuable.

Summary updated

BibTeX
@inproceedings{kim2025configuring,
  author = {Bum Jun Kim and Sang Woo Kim},
  title = {{Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i17.33962},
}

Loading related content...

Figure