Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization

Mainak Singha, Ankit Jha, Shirsha Bose, Ashwin Nair, Moloud Abdar, Biplab Banerjee

TL;DR

ODG-CLIP is the first vision-language framework designed for Open Domain Generalization (ODG), addressing both domain shifts and unseen categories at test time. By leveraging Stable Diffusion to synthesize domain-specific outlier images and introducing a novel prompt-driven latent visual space, it achieves an 8% to 16% performance boost over existing state-of-the-art methods.

What Problem Does This Paper Address?

Traditional Domain Generalization (DG) algorithms assume a "closed-set" scenario: a model is trained on multiple source domains (e.g., photos of dogs and cats) and must generalize to an unseen target domain (e.g., sketches of dogs and cats) containing the exact same classes.

Open Domain Generalization (ODG) introduces a much harsher, real-world twist. Not only does the model face a domain shift (photos to sketches), but it also encounters a category shift—the target domain contains entirely new, unseen "open-set" categories (e.g., a sketch of an elephant). Traditional CNN backbones struggle here because they lack the broad semantic open-world knowledge required to flag these novel outliers without prior training.

Methods

ODG-CLIP tackles this by building on top of the frozen vision and text encoders of CLIP, introducing three core innovations:

  1. Unified Multi-Class Classification via Stable Diffusion: Instead of using a binary "known vs. unknown" threshold, the authors treat ODG as a $C+1$ classification problem (where $C$ is the number of known classes, and $+1$ is the dedicated "unknown" class prompt). To train this unknown prompt, they use Stable Diffusion to generate realistic proxy outlier images ($D_{open}$). They use positive prompts like "a [domain] of an unknown class" paired with negative prompts listing all known classes to ensure the generated images mimic the source domain's style but contain completely distinct semantics.
  2. Domain-Aware Prompt Learning: The model learns two types of text prompts per image: one conditioned on both domain and class information, and another conditioned solely on the domain style. The domain token is dynamically computed using the mean and standard deviation of the image's visual features.
  3. Prompt-Driven Latent Visual Space: To enhance CLIP's visual embeddings, the framework subtracts the domain-only prompt embedding from the domain+class prompt embedding. This isolates pure, class-discriminative information ($\hat{x}_y$). This vector is then upsampled and combined with the original image to form a refined latent representation ($\tilde{x}^y$) for contrastive learning.
Model architecture of ODG-CLIP showing pseudo-open sample generation and prompt learning stages.

The entire framework is optimized using a standard contrastive loss ($\mathcal{L}_{con}$) alongside a novel Cross-Domain Semantic Consistency Loss ($\mathcal{L}_{sem}$), which forces the isolated semantic features of shared classes to align consistently across different domains.

Datasets

The authors evaluated ODG-CLIP across six diverse benchmark datasets using a strict leave-one-domain-out protocol:

  • PACS (Art, Cartoon, Sketch, Photo)
  • VLCS (PASCAL VOC, Caltech, LabelMe, Sun)
  • Office-Home (Art, Clipart, Product, Real)
  • Digits-DG (MNIST, MNIST-M, SVHN, SYN)
  • Multi-Dataset (Office-31, STL-10, Visda2017, DomainNet)
  • Mini-DomainNet and the full DomainNet (large-scale benchmarks)

Key Insights and Ideas

  • The Power of Negation: Standard generative methods for outlier detection (like GANs or image mix-ups) often produce semantically blurry or unrealistic images. By using conditional diffusion models with explicit negative prompts, ODG-CLIP generates highly diverse, high-quality outliers that perfectly match the source domain's texture and style.
  • Isolating Semantics via Subtraction: By explicitly subtracting the domain-only prompt from the domain+class prompt, the model mathematically disentangles "style" from "semantics." This prevents the visual encoder from overfitting to domain-specific noise.
  • Semantic Continuity: The introduction of $\mathcal{L}_{sem}$ ensures that a "dog" in a photo and a "dog" in a cartoon share a highly correlated latent representation, significantly reducing domain divergence.

Key Results

ODG-CLIP achieved new state-of-the-art performance across all evaluated closed-set and open-set settings:

  • Open Domain Generalization: As shown in the comprehensive evaluation table, ODG-CLIP outperformed the best CNN-based method (MEDIC) by 30.54% and zero-shot CLIP by 21.44% in average H-score (harmonic mean of closed and open-set accuracy).
  • Closed-Set Domain Generalization: Even when no category shift was present, ODG-CLIP dominated, outperforming traditional DG methods like SWAD and prompt-learning baselines like MaPLe and StyLIP.
Closed-set DG performance comparison showing ODG-CLIP's superiority.
  • Outlier Quality: The proxy images generated via Stable Diffusion achieved an Inception Score of 6.67, vastly superior to traditional OSR data-generation techniques like CuMix (1.18) and OpenGAN (1.43), leading to a much more robust outlier classifier.
Figure 4. Comparison of ODG-CLIP with Cumix [51] and Open- GAN [36] on inception score for the generated open-set samples.

Limitations and Future Work

While ODG-CLIP excels at image classification tasks under domain and category shifts, the authors note that the framework is currently limited to global image-level tasks. A natural and promising future direction is extending this prompt-driven latent space refinement to dense prediction tasks, such as open-domain object detection and semantic segmentation.

Summary updated

Abstract

We delve into Open Domain Generalization (ODG), marked by domain and category shifts between training's labeled source and testing's unlabeled target domains. Existing solutions to ODG face limitations due to constrained generalizations of traditional CNN backbones and errors in detecting target open samples in the absence of prior knowledge. Addressing these pitfalls, we introduce ODG-CLIP, harnessing the semantic prowess of the vision-language model, CLIP. Our framework brings forth three primary innovations: Firstly, distinct from prevailing paradigms, we conceptualize ODG as a multi-class classification challenge encompassing both known and novel categories. Central to our approach is modeling a unique prompt tailored for detecting unknown class samples, and to train this, we employ a readily accessible stable diffusion model, elegantly generating proxy images for the open class. Secondly, aiming for domain-tailored classification (prompt) weights while ensuring a balance of precision and simplicity, we devise a novel visual stylecentric prompt learning mechanism. Finally, we infuse images with class-discriminative knowledge derived from the prompt space to augment the fidelity of CLIP's visual embeddings. We introduce a novel objective to safeguard the continuity of this infused semantic intel across domains, especially for the shared classes. Through rigorous testing on diverse datasets, covering closed and open-set DG contexts, ODG-CLIP demonstrates clear supremacy, consistently outpacing peers with performance boosts between 8%-16%. Code will be available at https://github.com/mainaksingha01/ODG-CLIP.
BibTeX
@inproceedings{singha2024unknown,
  author = {Mainak Singha and Ankit Jha and Shirsha Bose and Ashwin Nair and Moloud Abdar and Biplab Banerjee},
  title = {{Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=FY6AQKgw2B},
}
Figure