MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations

Reza Akbarian Bafghi, Nidhin Harilal, Maziar Raissi, Claire MonteleoniWACV 2025

TL;DR

This paper introduces MixDiff, a novel self-supervised learning (SSL) framework that combines real and synthetic images to train more robust visual representations. By replacing one augmented view of a real image with a synthetically generated counterpart, MixDiff significantly boosts performance on various downstream tasks, including those involving distribution shifts, and achieves comparable results with less real data.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has made remarkable progress in learning powerful visual representations without relying on labeled data. However, current SSL methods predominantly use real images, which can be costly to collect and annotate, and may not always capture the full spectrum of visual variability. While synthetic images offer an attractive alternative due to their ease of generation and diversity, models trained solely on them often suffer from a "domain gap," leading to poor performance on real-world data. This paper addresses the challenge of bridging this domain gap and leveraging the strengths of both real and synthetic data to achieve more robust and adaptable SSL representations.

Methods

The core innovation of MixDiff lies in its simple yet effective integration of synthetic images into existing joint-embedding SSL frameworks. The framework works by taking a standard SSL pipeline (like SimCLR, BarlowTwins, or DINO) and replacing one of the augmented views of a real image with a synthetic image generated by a diffusion model (specifically, a fine-tuned Stable Diffusion variant). This modification allows the model to learn cross real-synthetic image representations. The key idea is that synthetically generated images act as "harder positive pairs," making the training objective less trivial and forcing the model to learn more generalized features. MixDiff is designed to be plug-and-play, easily incorporated into various SSL architectures without requiring significant changes to their core mechanisms.

Datasets

The paper utilizes the following datasets:

  • Training: ImageNet-100 (IN-100) for most experiments due to its manageable size, and ImageNet-1K (IN-1K) for validation on a larger scale.
  • Evaluation (Distribution Shifts): ImageNet-A (IN-A), ImageNet-Sketch (IN-Sketch), ObjectNet, ImageNet-V2 (IN-V2), VizWiz-Classification (VW-C), and ImageNet-R (IN-R).
  • Evaluation (Transfer Learning): CIFAR-10, CIFAR-100, Aircraft, DTD, Flowers102, Food101, Pets37, and STL10.

No new benchmarks were introduced; the focus was on evaluating performance on existing datasets under various conditions.

Key Insights and Ideas

  • Synthetic data alone is insufficient: Models trained exclusively on synthetic images generally underperform those trained on real images, highlighting the persistent domain gap.
  • Mixing is key: Combining real and synthetic images (MixDiff) leads to significantly more robust and adaptable representations than using either data source alone.
  • Harder positive pairs: Synthetic images act as more challenging positive examples, preventing models from learning trivial shortcuts and encouraging the learning of more generalized features.
  • Robustness to synthetic image quality: MixDiff shows less sensitivity to variations in synthetic image quality, reducing the need for precise optimization of generative models.
  • Reduced data requirement: MixDiff achieves comparable performance to models trained on full real datasets while using a significantly smaller proportion of real data, demonstrating efficiency.
  • Augmentation independence: Surprisingly, MixDiff can achieve strong performance even with reduced or no data augmentations, a critical component in traditional SSL.

Key Results

  • ImageNet-1K Accuracy: MixDiff enhances SimCLR's ImageNet-1K accuracy by 4.56%.
  • Distribution Shift Robustness: MixDiff models show an average increase in top-1 accuracy of about 26.92% across six distributional datasets.
  • Transfer Learning: MixDiff achieves an average improvement of 7.36% in transfer learning across eight diverse datasets.
  • Data Efficiency: MixDiff achieves comparable performance to SimCLR trained on 100% real data while using only 53% (for in-distribution) and 67% (for robustness) of the real data.
  • Model Agnostic: MixDiff successfully enhances SimCLR, BarlowTwins, and DINO.

Limitations and Future Work

  • Computational Investment: While MixDiff offers data efficiency, it requires an initial computational investment for generating synthetic images. The authors mention exploring strategies to reduce training times with minimal performance impact.
  • Generative Model Exploration: The paper primarily uses Stable Diffusion. Future work could investigate advanced generative diffusion models to further improve mixing and enhance the robustness of visual features.
  • Understanding the "Why": While MixDiff shows robustness to synthetic image quality variations, further research is needed to fully understand this phenomenon and its implications for representation learning.
  • Supervised vs. SSL Gap: While MixDiff narrows the gap between SSL and supervised models, a performance gap still exists, suggesting room for further improvement in SSL techniques.

Summary updated

BibTeX
@inproceedings{bafghi2025mixdiff,
  author = {Reza Akbarian Bafghi and Nidhin Harilal and Maziar Raissi and Claire Monteleoni},
  title = {{MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure