TL;DR
ODG-CLIP is the first vision-language framework designed for Open Domain Generalization (ODG), addressing both domain shifts and unseen categories at test time. By leveraging Stable Diffusion to synthesize domain-specific outlier images and introducing a novel prompt-driven latent visual space, it achieves an 8% to 16% performance boost over existing state-of-the-art methods.
What Problem Does This Paper Address?
Traditional Domain Generalization (DG) algorithms assume a "closed-set" scenario: a model is trained on multiple source domains (e.g., photos of dogs and cats) and must generalize to an unseen target domain (e.g., sketches of dogs and cats) containing the exact same classes.
Open Domain Generalization (ODG) introduces a much harsher, real-world twist. Not only does the model face a domain shift (photos to sketches), but it also encounters a category shift—the target domain contains entirely new, unseen "open-set" categories (e.g., a sketch of an elephant). Traditional CNN backbones struggle here because they lack the broad semantic open-world knowledge required to flag these novel outliers without prior training.
Methods
ODG-CLIP tackles this by building on top of the frozen vision and text encoders of CLIP, introducing three core innovations:
- Unified Multi-Class Classification via Stable Diffusion: Instead of using a binary "known vs. unknown" threshold, the authors treat ODG as a $C+1$ classification problem (where $C$ is the number of known classes, and $+1$ is the dedicated "unknown" class prompt). To train this unknown prompt, they use Stable Diffusion to generate realistic proxy outlier images ($D_{open}$). They use positive prompts like
"a [domain] of an unknown class"paired with negative prompts listing all known classes to ensure the generated images mimic the source domain's style but contain completely distinct semantics. - Domain-Aware Prompt Learning: The model learns two types of text prompts per image: one conditioned on both domain and class information, and another conditioned solely on the domain style. The domain token is dynamically computed using the mean and standard deviation of the image's visual features.
- Prompt-Driven Latent Visual Space: To enhance CLIP's visual embeddings, the framework subtracts the domain-only prompt embedding from the domain+class prompt embedding. This isolates pure, class-discriminative information ($\hat{x}_y$). This vector is then upsampled and combined with the original image to form a refined latent representation ($\tilde{x}^y$) for contrastive learning.
The entire framework is optimized using a standard contrastive loss ($\mathcal{L}_{con}$) alongside a novel Cross-Domain Semantic Consistency Loss ($\mathcal{L}_{sem}$), which forces the isolated semantic features of shared classes to align consistently across different domains.
Datasets
The authors evaluated ODG-CLIP across six diverse benchmark datasets using a strict leave-one-domain-out protocol:
- PACS (Art, Cartoon, Sketch, Photo)
- VLCS (PASCAL VOC, Caltech, LabelMe, Sun)
- Office-Home (Art, Clipart, Product, Real)
- Digits-DG (MNIST, MNIST-M, SVHN, SYN)
- Multi-Dataset (Office-31, STL-10, Visda2017, DomainNet)
- Mini-DomainNet and the full DomainNet (large-scale benchmarks)
Key Insights and Ideas
- The Power of Negation: Standard generative methods for outlier detection (like GANs or image mix-ups) often produce semantically blurry or unrealistic images. By using conditional diffusion models with explicit negative prompts, ODG-CLIP generates highly diverse, high-quality outliers that perfectly match the source domain's texture and style.
- Isolating Semantics via Subtraction: By explicitly subtracting the domain-only prompt from the domain+class prompt, the model mathematically disentangles "style" from "semantics." This prevents the visual encoder from overfitting to domain-specific noise.
- Semantic Continuity: The introduction of $\mathcal{L}_{sem}$ ensures that a "dog" in a photo and a "dog" in a cartoon share a highly correlated latent representation, significantly reducing domain divergence.
Key Results
ODG-CLIP achieved new state-of-the-art performance across all evaluated closed-set and open-set settings:
- Open Domain Generalization: As shown in the comprehensive evaluation table, ODG-CLIP outperformed the best CNN-based method (MEDIC) by 30.54% and zero-shot CLIP by 21.44% in average H-score (harmonic mean of closed and open-set accuracy).
- Closed-Set Domain Generalization: Even when no category shift was present, ODG-CLIP dominated, outperforming traditional DG methods like SWAD and prompt-learning baselines like MaPLe and StyLIP.
- Outlier Quality: The proxy images generated via Stable Diffusion achieved an Inception Score of 6.67, vastly superior to traditional OSR data-generation techniques like CuMix (1.18) and OpenGAN (1.43), leading to a much more robust outlier classifier.
Limitations and Future Work
While ODG-CLIP excels at image classification tasks under domain and category shifts, the authors note that the framework is currently limited to global image-level tasks. A natural and promising future direction is extending this prompt-driven latent space refinement to dense prediction tasks, such as open-domain object detection and semantic segmentation.
Summary updated