TL;DR
This paper introduces Gen-SIS, a novel self-supervised learning framework that leverages generative diffusion models for data augmentation. By training a diffusion model solely on unlabeled image data and its own learned representations, Gen-SIS generates diverse "self-augmentations" that significantly boost the performance of visual encoders on various downstream tasks, including those in specialized domains like histopathology.
What Problem Does This Paper Address?
Self-supervised learning (SSL) methods excel at learning visual representations from unlabeled data by training models to recognize different "views" of the same image. Traditionally, these views are created using hand-crafted augmentations like random cropping and color jittering. While effective, these augmentations can be limited in their diversity. Recent work has explored using generative diffusion models to create richer augmentations. However, these models often require large-scale, pre-trained image-text datasets, which are not always available for specialized domains like histopathology. Furthermore, adapting these models can be challenging and may introduce unwanted biases from the pre-training data. Gen-SIS aims to bridge this gap by enabling the use of diffusion-based augmentations in a completely self-supervised manner, without relying on external text supervision or pre-trained models, making it applicable to a wider range of domains.
Methods
Gen-SIS consists of two main steps:
- Embedding-Conditioned Latent Diffusion Model (E-LDM): The core idea is to train a diffusion model that generates synthetic images conditioned on the embeddings extracted from a pre-trained SSL encoder. This E-LDM is trained exclusively on the unlabeled dataset, meaning it learns to generate variations of images based on the representations learned by the SSL encoder itself. This is the "self-augmentation" part.
- Generative Augmentations: The E-LDM can synthesize new images from a single source image's embedding, creating variations in shape, size, and background while preserving semantic content.
- Interpolated Augmentations: The E-LDM can also interpolate between the embeddings of two source images to generate a synthetic image that semantically blends features from both.
- Enhanced SSL Training: The generated "self-augmentations" are then used to train a stronger SSL encoder.
- View Invariance: Real images and their corresponding generative augmentations are treated as different views for the standard SSL pretext task (e.g., DINO's self-distillation).
- Disentanglement Pretext Task: For interpolated augmentations, a novel pretext task is introduced. The model is trained to disentangle the features of the two original source images from the interpolated synthetic image. This encourages the encoder to learn finer-grained, discriminative features.
Datasets
- Training & Evaluation (Natural Images): ImageNet-1K.
- Evaluation (Natural Images): ImageNet-A, ImageNet-R, ImageNet-Sketch (for robustness), Oxford and Paris datasets (for image retrieval), INRIA Copydays dataset (for copy detection), DAVIS-2017 (for video object segmentation).
- Training & Evaluation (Histopathology): PANDA (prostate cancer WSIs), BRIGHT (breast cancer WSIs).
Key Insights and Ideas
- Self-Supervised Diffusion: The most significant insight is that diffusion models can be effectively trained for augmentation without external text supervision, by conditioning them on the representations learned by an SSL encoder trained on the same unlabeled data. This unlocks diffusion models for domains lacking large image-text datasets.
- "Self-Augmentation" Power: Generative augmentations, when trained in a self-supervised manner, provide a richer set of views than traditional hand-crafted augmentations, leading to improved SSL performance.
- Interpolation for Disentanglement: The ability of diffusion models to interpolate in the latent space is cleverly leveraged to create a novel pretext task. This "disentanglement" task forces the model to learn to separate and understand distinct features from multiple sources, significantly boosting representation quality.
- Domain Generalization: The framework demonstrates strong performance not only on object-centric natural images but also on context-based histopathology images, highlighting its broad applicability.
Key Results
- ImageNet-1K: Gen-DINO (Gen-SIS applied to DINO) achieved a 1.5% improvement in Top-1% accuracy on k-NN evaluation compared to vanilla DINO.
- Robustness: Gen-DINO showed significant improvements on out-of-distribution datasets like ImageNet-R (+4.73% accuracy) and Sketch (+0.37% accuracy).
- Image Retrieval & Copy Detection: Gen-DINO features outperformed DINO features by up to 2.6% in mAP for image retrieval and improved copy detection performance by 2.3%.
- Video Object Segmentation: Gen-DINO showed improved performance on the DAVIS-2017 benchmark, particularly in mean region similarity (Im) and mean contour-based accuracy (Fm).
- Histopathology: Gen-DINO improved balanced accuracy on the PANDA dataset by over 3% and accuracy on the BRIGHT dataset by 1.7%, demonstrating its effectiveness in non-object-centric domains.
- Disentanglement Importance: Ablations showed that the disentanglement pretext task was crucial, contributing a larger portion of the performance gains than generative augmentation alone.
Limitations and Future Work
- Interpolation Ratio: The paper notes that the choice of interpolation ratio ($\alpha$) for interpolated augmentations can impact performance. While $\alpha=0.5$ generally works best, the authors observed that for highly diverse datasets like ImageNet-1K, the diffusion model might struggle to perfectly blend features at extreme $\alpha$ values, leading to noisy training.
- Future Work: The authors suggest exploring more flexible interpolation augmentation strategies, including policies for selecting which image pairs to interpolate. They also aim to scale Gen-SIS with larger datasets.
Summary updated