Pasta: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization

Prithvijit Chattopadhyay, Kartik Sarangmath, Vivek Vijaykumar, Judy Hoffman

Abstract

Synthetic data offers the promise of cheap and bountiful training data for settings where labeled real-world data is scarce. However, models trained on synthetic data significantly underperform when evaluated on real-world data. In this paper, we propose Proportional Amplitude Spectrum Training Augmentation (Pasta), a simple and effective augmentation strategy to improve out-of-the-box synthetic-to-real (syn-to-real) generalization performance. Pasta perturbs the amplitude spectra of synthetic images in the Fourier domain to generate augmented views. Specifically, with Pasta we propose a structured perturbation strategy where high-frequency components are perturbed relatively more than the low-frequency ones. For the tasks of semantic segmentation (GTAV→Real), object detection (Sim10K→Real), and object recognition (VisDA-C Syn→Real), across a total of 5 syn-to-real shifts, we find that Pasta outperforms more complex state-of-the-art generalization methods while being complementary to the same.

Introduction

Deep learning models thrive on massive labeled datasets, but labeling real-world images is expensive and slow. Synthetic data, often sourced from high-fidelity game engines like GTA V, offers a "cheap" alternative. However, models trained on such data often fail to generalize to real-world environments—a challenge known as the syn-to-real domain gap.

The authors identify a specific reason for this failure: synthetic images are rendered with much less diversity in their high-frequency (HF) details compared to real-world captures. Real images contain complex sensor noise, atmospheric interference, and intricate textures that simulators often oversimplify. Consequently, models overfit to the "clean" high-frequency patterns of synthetic data, making them brittle when they encounter the messy reality of a real camera feed.

Methodology

PASTA operates in the Fourier Domain rather than the standard pixel space. The process follows these steps:

  1. Decomposition: An input synthetic image is transformed using a 2D Fast Fourier Transform (FFT) into two components: the Phase Spectrum (which captures high-level semantics and shapes) and the Amplitude Spectrum (which captures low-level statistics, lighting, and texture).
  2. Proportional Perturbation: The authors keep the phase spectrum pristine to preserve the image's meaning. However, they apply multiplicative "jitter" to the amplitude spectrum. Crucially, the strength of this jitter is proportional to the spatial frequency. High-frequency components (fine details) are perturbed significantly more than low-frequency components (global structure).
  3. Reconstruction: The perturbed amplitude is combined with the original phase, and an Inverse FFT (iFFT) is applied to create the augmented "PASTA" image.
Comparison of original synthetic images with RandAug and PASTA augmentations.

This approach is "plug-and-play," meaning it can be added to any existing training pipeline (ResNet, SegFormer, etc.) as a simple data preprocessing step.

Experimental Results

The authors evaluated PASTA across three major computer vision tasks using standard "syn-to-real" benchmarks:

  • Semantic Segmentation: Trained on GTAV; evaluated on Cityscapes, BDD100K, and Mapillary Vistas. On the GTAV → Cityscapes shift, a standard DeepLabv3+ model improved from 28.95 mIoU to 44.12 mIoU—a staggering 15+ point absolute increase.
  • Object Detection: Trained on Sim10K (synthetic cars); evaluated on Cityscapes. PASTA provided a +16.9 mAP improvement over the baseline, outperforming more complex domain generalization methods.
  • Object Recognition: Trained on VisDA-C (synthetic 3D models); evaluated on real-world object crops.

Key Insights

  • The HF Variance Gap: Analysis of amplitude values across datasets revealed that real-world datasets (like Cityscapes) have significantly higher variance in high-frequency bands than synthetic ones (like GTAV). PASTA essentially "injects" this missing variance.
  • Phase is Sacred: PASTA demonstrates that keeping the phase untouched is vital for maintaining semantic integrity, while the amplitude is where the "style" and "domain" information resides.
  • Monotonicity Matters: The authors confirmed that perturbing high frequencies more than low frequencies is the correct strategy; reversing this actually hurts performance.
  • Complementary Nature: PASTA doesn't replace standard augmentations like ColorJitter or RandAugment; it works alongside them. Because it operates in the frequency domain, it captures "nuisance" characteristics that pixel-level operations miss.

Conclusion

PASTA provides a simple yet powerful Fourier-domain augmentation strategy designed to bridge the syn-to-real gap. By disproportionately perturbing the high-frequency components of synthetic images during training, PASTA forces models to become invariant to the "too-perfect" details of simulated data. The method is versatile, effective across diverse architectures (CNNs and Transformers), and achieves state-of-the-art results for out-of-the-box generalization without requiring any real-world images during training.

Summary updated

Abstract

Synthetic data offers the promise of cheap and bountiful training data for settings where labeled real-world data is scarce. However, models trained on synthetic data significantly underperform when evaluated on real-world data. In this paper, we propose Proportional Amplitude Spectrum Training Augmentation (Pasta), a simple and effective augmentation strategy to improve out-of-the-box synthetic-to-real (syn-to-real) generalization performance. Pasta perturbs the amplitude spectra of synthetic images in the Fourier domain to generate augmented views. Specifically, with Pasta we propose a structured perturbation strategy where high-frequency components are perturbed relatively more than the low-frequency ones. For the tasks of semantic segmentation (GTAV→Real), object detection (Sim10K→Real), and object recognition (VisDA-C Syn→Real), across a total of 5 syn-to-real shifts, we find that Pasta outperforms more complex state-of-the-art generalization methods while being complementary to the same.
BibTeX
@misc{chattopadhyay2022pasta,
  author = {Prithvijit Chattopadhyay and Kartik Sarangmath and Vivek Vijaykumar and Judy Hoffman},
  title = {{Pasta: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization}},
  year = {2022},
  eprint = {2212.00979},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.00979},
}
Figure