What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalization

Xavier Thomas, Deepti GhadiyaramICCV 2025

TL;DR

This paper introduces GUIDE, a novel framework that enhances domain generalization by leveraging the rich latent representations learned by diffusion models. By discovering and incorporating "pseudo-domains" from these latent spaces, GUIDE significantly improves a model's ability to generalize to unseen data distributions without requiring domain labels during training.

What Problem Does This Paper Address?

In machine learning, a persistent challenge is domain generalization: building models that perform well not just on data similar to what they were trained on, but also on entirely new, unseen data distributions. This is crucial because real-world data often varies due to factors like different cameras, lighting conditions, or artistic styles. Traditional methods often struggle when test data deviates from training data, even subtly. While large pre-trained models (like those from diffusion models) offer powerful feature representations, understanding what these representations capture and how they can be best utilized for domain generalization remains an open question. Specifically, this paper investigates how different pre-training objectives influence the latent space's ability to capture domain-specific variations and proposes a method to leverage these variations for better generalization.

Methods

The core of the proposed method, GUIDE (Generalization using Inferred Domains from Latent Embeddings), involves two main steps:

  1. Unsupervised Pseudo-Domain Discovery: Given a pre-trained feature extractor (denoted as $\Psi$), the method first clusters the extracted features from the training data. These clusters, termed "pseudo-domains," represent underlying domain-specific variations without explicit domain labels. The centroid of each cluster serves as a compact representation of that pseudo-domain.
  2. Augmenting Classifier Representations: The discovered pseudo-domain representations are then used to augment the standard input features ($\Phi(x)$) fed into a classifier. A transformation function ($T$) is applied to align the pseudo-domain representations with the main feature space, mitigating potential domain drift. The concatenated features are then used to train a classifier. This approach aims to provide the classifier with complementary information about potential domain variations, making it more robust.

Crucially, GUIDE operates without requiring domain labels during training or testing, making it applicable in scenarios where such labels are unavailable or noisy.

Datasets

The paper evaluates GUIDE on several standard domain generalization benchmarks:

  • PACS [35]
  • VLCS [17]
  • TerraIncognita [4] (TI)
  • OfficeHome [70] (OH)
  • DomainNet [52] (DN)

Additionally, they introduce two synthetic datasets:

  • Synth-Photography: Features photographic styles like macro, tilt-shift, and bokeh.
  • Synth-Artists: Captures artistic styles from painters like Van Gogh and Kinkade.

These synthetic datasets were generated using Stable Diffusion XL [53] to provide controlled variations in photographic and artistic styles.

Key Insights and Ideas

  • Diffusion Models Excel at Domain Separation: The paper's key insight is that diffusion models, due to their generative, class-agnostic training objective, naturally learn latent spaces that effectively separate domain-specific variations (like style, texture, camera angles) from class-specific information. This is a significant advantage over discriminative models that prioritize class separation.
  • Pseudo-Domains as Auxiliary Information: The success of GUIDE hinges on the idea that augmenting a classifier's primary features with these discovered "pseudo-domain" representations provides valuable, complementary information. This auxiliary guidance helps the model become more resilient to shifts in data distribution.
  • Layer Choice Matters for Diffusion Models: Even within diffusion models, the specific layer from which features are extracted significantly impacts their ability to capture domain-specific information. Transformer-based DiT models seem better at global context and style separation (e.g., on PACS), while U-Net based SD-2.1 excels at fine-grained spatial details (e.g., on TerraIncognita).
  • Domain NMI as a Predictive Metric: The Normalized Mutual Information (NMI) between discovered clusters and true domain labels (domain NMI) proves to be a strong indicator of a feature space's suitability for domain generalization. Higher domain NMI generally correlates with better generalization performance.

Key Results

  • Significant Accuracy Gains: GUIDE, particularly when using diffusion model features (DiT and SD-2.1), consistently outperforms the standard Empirical Risk Minimization (ERM) baseline. On average, GUIDE achieves a +1.9% improvement over ERM, with notable gains on specific datasets like VLCS (+1.9%) and PACS (+3.3%) for DiT, and TerraIncognita (+4.3%) for SD-2.1.
  • State-of-the-Art Performance: When compared against other domain generalization algorithms, GUIDE-BEST (an ensemble of the best diffusion features) achieves the highest average performance of 66.3% without using domain labels.
  • Diffusion Features Superiority: Across multiple datasets, diffusion model features (DiT, SD-2.1) consistently yield higher accuracy gains compared to traditional vision models like ResNet-50, CLIP, DINOv2, and MAE.
  • Style Discovery: The method effectively separates artistic and photographic styles, demonstrating its utility for understanding and curating datasets. DiT showed strong performance on artistic styles (Synth-Artists), while SD-2.1 was better for photographic styles (Synth-Photography).

Limitations and Future Work

The authors acknowledge a few limitations:

  • Hyperparameter Selection: The optimal number of pseudo-domains (K) is a hyperparameter, and the paper uses a heuristic from prior work. Developing a more dataset-specific or adaptive method for selecting K would be beneficial.
  • Computational Overhead: Using an additional feature extractor and the transformation function adds some computational cost.
  • Combining Models: Future work could explore combining features from multiple models or different layers to build a more universally robust latent space that works "out of the box" for diverse tasks.

Summary updated

BibTeX
@inproceedings{thomas2025what,
  author = {Xavier Thomas and Deepti Ghadiyaram},
  title = {{What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure