Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning

Varun Belagali, Srikar Yellapragada, Alexandros Graikos, Saarthak Kapse, Zilinghan Li, Tarak Nath Nandi, Ravi K. Madduri, Prateek Prasanna, Joel H. Saltz, Dimitris SamarasarXiv 2024 · DOI · OpenReview

TL;DR

This paper introduces Gen-SIS, a novel self-supervised learning framework that leverages generative diffusion models for data augmentation. By training a diffusion model solely on unlabeled image data and its own learned representations, Gen-SIS generates diverse "self-augmentations" that significantly boost the performance of visual encoders on various downstream tasks, including those in specialized domains like histopathology.

What Problem Does This Paper Address?

Self-supervised learning (SSL) methods excel at learning visual representations from unlabeled data by training models to recognize different "views" of the same image. Traditionally, these views are created using hand-crafted augmentations like random cropping and color jittering. While effective, these augmentations can be limited in their diversity. Recent work has explored using generative diffusion models to create richer augmentations. However, these models often require large-scale, pre-trained image-text datasets, which are not always available for specialized domains like histopathology. Furthermore, adapting these models can be challenging and may introduce unwanted biases from the pre-training data. Gen-SIS aims to bridge this gap by enabling the use of diffusion-based augmentations in a completely self-supervised manner, without relying on external text supervision or pre-trained models, making it applicable to a wider range of domains.

Methods

Gen-SIS consists of two main steps:

  1. Embedding-Conditioned Latent Diffusion Model (E-LDM): The core idea is to train a diffusion model that generates synthetic images conditioned on the embeddings extracted from a pre-trained SSL encoder. This E-LDM is trained exclusively on the unlabeled dataset, meaning it learns to generate variations of images based on the representations learned by the SSL encoder itself. This is the "self-augmentation" part.
  2. Generative Augmentations: The E-LDM can synthesize new images from a single source image's embedding, creating variations in shape, size, and background while preserving semantic content.
  3. Interpolated Augmentations: The E-LDM can also interpolate between the embeddings of two source images to generate a synthetic image that semantically blends features from both.
  4. Enhanced SSL Training: The generated "self-augmentations" are then used to train a stronger SSL encoder.
  5. View Invariance: Real images and their corresponding generative augmentations are treated as different views for the standard SSL pretext task (e.g., DINO's self-distillation).
  6. Disentanglement Pretext Task: For interpolated augmentations, a novel pretext task is introduced. The model is trained to disentangle the features of the two original source images from the interpolated synthetic image. This encourages the encoder to learn finer-grained, discriminative features.

Datasets

  • Training & Evaluation (Natural Images): ImageNet-1K.
  • Evaluation (Natural Images): ImageNet-A, ImageNet-R, ImageNet-Sketch (for robustness), Oxford and Paris datasets (for image retrieval), INRIA Copydays dataset (for copy detection), DAVIS-2017 (for video object segmentation).
  • Training & Evaluation (Histopathology): PANDA (prostate cancer WSIs), BRIGHT (breast cancer WSIs).

Key Insights and Ideas

  • Self-Supervised Diffusion: The most significant insight is that diffusion models can be effectively trained for augmentation without external text supervision, by conditioning them on the representations learned by an SSL encoder trained on the same unlabeled data. This unlocks diffusion models for domains lacking large image-text datasets.
  • "Self-Augmentation" Power: Generative augmentations, when trained in a self-supervised manner, provide a richer set of views than traditional hand-crafted augmentations, leading to improved SSL performance.
  • Interpolation for Disentanglement: The ability of diffusion models to interpolate in the latent space is cleverly leveraged to create a novel pretext task. This "disentanglement" task forces the model to learn to separate and understand distinct features from multiple sources, significantly boosting representation quality.
  • Domain Generalization: The framework demonstrates strong performance not only on object-centric natural images but also on context-based histopathology images, highlighting its broad applicability.

Key Results

  • ImageNet-1K: Gen-DINO (Gen-SIS applied to DINO) achieved a 1.5% improvement in Top-1% accuracy on k-NN evaluation compared to vanilla DINO.
  • Robustness: Gen-DINO showed significant improvements on out-of-distribution datasets like ImageNet-R (+4.73% accuracy) and Sketch (+0.37% accuracy).
  • Image Retrieval & Copy Detection: Gen-DINO features outperformed DINO features by up to 2.6% in mAP for image retrieval and improved copy detection performance by 2.3%.
  • Video Object Segmentation: Gen-DINO showed improved performance on the DAVIS-2017 benchmark, particularly in mean region similarity (Im) and mean contour-based accuracy (Fm).
  • Histopathology: Gen-DINO improved balanced accuracy on the PANDA dataset by over 3% and accuracy on the BRIGHT dataset by 1.7%, demonstrating its effectiveness in non-object-centric domains.
  • Disentanglement Importance: Ablations showed that the disentanglement pretext task was crucial, contributing a larger portion of the performance gains than generative augmentation alone.

Limitations and Future Work

  • Interpolation Ratio: The paper notes that the choice of interpolation ratio ($\alpha$) for interpolated augmentations can impact performance. While $\alpha=0.5$ generally works best, the authors observed that for highly diverse datasets like ImageNet-1K, the diffusion model might struggle to perfectly blend features at extreme $\alpha$ values, leading to noisy training.
  • Future Work: The authors suggest exploring more flexible interpolation augmentation strategies, including policies for selecting which image pairs to interpolate. They also aim to scale Gen-SIS with larger datasets.

Summary updated

Abstract

Self-supervised learning (SSL) methods have emerged as strong visual representation learners by training an image encoder to maximize similarity between features of different views of the same image. To perform this view-invariance task, current SSL algorithms rely on hand-crafted augmentations such as random cropping and color jittering to create multiple views of an image. Recently, generative diffusion models have been shown to improve SSL by providing a wider range of data augmentations. However, these diffusion models require pre-training on large-scale image-text datasets, which might not be available for many specialized domains like histopathology. In this work, we introduce Gen-SIS, a diffusion-based augmentation technique trained exclusively on unlabeled image data, eliminating any reliance on external sources of supervision such as text captions. We first train an initial SSL encoder on a dataset using only hand-crafted augmentations. We then train a diffusion model conditioned on embeddings from that SSL encoder. Following training, given an embedding of the source image, this diffusion model can synthesize its diverse views. We show that these `self-augmentations', i.e. generative augmentations based on the vanilla SSL encoder embeddings, facilitate the training of a stronger SSL encoder. Furthermore, based on the ability to interpolate between images in the encoder latent space, we introduce the novel pretext task of disentangling the two source images of an interpolated synthetic image. We validate Gen-SIS's effectiveness by demonstrating performance improvements across various downstream tasks in both natural images, which are generally object-centric, as well as digital histopathology images, which are typically context-based.
BibTeX
@misc{belagali2024gen,
  author = {Varun Belagali and Srikar Yellapragada and Alexandros Graikos and Saarthak Kapse and Zilinghan Li and Tarak Nath Nandi and Ravi K. Madduri and Prateek Prasanna and Joel H. Saltz and Dimitris Samaras},
  title = {{Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning}},
  year = {2024},
  eprint = {2412.01672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01672},
}
Figure