Abstract
Domain Generalization (DG) is a critical challenge in computer vision, where models must perform accurately on data from unseen distributions. While Vision Foundation Models (VFMs) like CLIP offer strong zero-shot capabilities, standard prompt tuning often causes models to overfit to the specific styles of the training domains. We introduce Disentangled Prompt Representation (DPR), a framework that leverages the semantic knowledge of Large Language Models (LLMs) to decouple object content from domain-specific styles. By using GPT-3 to generate distinct descriptions for invariant object features and domain-specific attributes, DPR guides the visual encoder to learn more robust representations. Our approach achieves new state-of-the-art results across five major DG benchmarks, including a significant improvement on the large-scale DomainNet dataset.
Introduction
A fundamental assumption in traditional machine learning is that training and testing data share the same distribution. However, real-world applications frequently encounter "domain shift"—for instance, a model trained on photographs of sheep may struggle to identify a sheep in a charcoal sketch or a stylized painting. This challenge, known as Domain Generalization (DG), requires models to distinguish between the essential features of an object (domain-invariant) and its stylistic presentation (domain-specific).
Current methods using Vision Foundation Models like CLIP often rely on simple prompts (e.g., "a photo of a [class]"). While effective for zero-shot tasks, fine-tuning these prompts can lead the model to conflate object identity with training-specific styles, hindering generalization to new domains. The core difficulty lies in disentanglement: teaching a model to separate the "sheep-ness" from the "sketch-ness."
Methodology
The DPR framework addresses the disentanglement problem through a three-stage process:
1. GPT-Assisted Text-Disentanglement
The core innovation lies in using the semantic knowledge of LLMs to define the boundaries between content and style. Instead of generic prompts, we query GPT-3 to provide detailed descriptions of what an object looks like generally (domain-invariant) and how it appears in specific domains like "sketch" or "cartoon" (domain-specific). This creates a semantic roadmap for the vision model to follow.
2. Image-Disentanglement Guided by Text
Using these disentangled text descriptions as anchors, we employ Visual Prompt Tuning (VPT) to train two distinct visual pathways. One pathway is optimized to align with domain-invariant text, capturing the "essence" of the category. The second pathway aligns with domain-specific text, learning to represent the stylistic nuances of the training domains.
3. Domain-Specific Prototype Learning (DSPL)
Rather than discarding style information during inference, DPR maintains a memory bank of prototypes from known domains. When presented with an image from an unseen domain, the model identifies the most similar known styles. It then combines a "style-aware" prediction with the "style-agnostic" prediction to produce a final classification, effectively leveraging its prior knowledge of different artistic or photographic styles.
Experimental Evaluation
We evaluated DPR on five standard DG benchmarks: PACS, VLCS, OfficeHome, DomainNet, and TerraInc. These datasets cover a wide range of domain shifts, from artistic styles to geographic variations in camera trap images.
Key Results
DPR established new state-of-the-art (SOTA) performance across all benchmarks. Notably:
- DomainNet: Achieved 62.05% accuracy, a significant increase over the previous SOTA of 59.80%.
- PACS: Reached 97.45% accuracy.
- Data Efficiency: The framework performs exceptionally well in "few-shot" settings, requiring only 16 images per class for training.
Ablation studies confirmed that both the text-based disentanglement and the prototype-based inference are essential for these gains. Removing either component led to a measurable drop in generalization performance.
Conclusion
The Disentangled Prompt Representation (DPR) framework demonstrates that LLMs can serve as powerful semantic teachers for vision models. By explicitly "unmixing" content from style in the text modality, we can guide vision models toward more robust, domain-invariant representations. While the method introduces some computational overhead and a dependency on external LLMs, its superior performance on large-scale benchmarks like DomainNet highlights the potential of disentangled representations for solving the domain generalization challenge.
Summary updated