Disentangled Prompt Representation for Domain Generalization

De Cheng, Zhipeng Xu, Xinyang Jiang, Nannan Wang, Dongsheng Li, Xinbo Gao

Abstract

Domain Generalization (DG) is a critical challenge in computer vision, where models must perform accurately on data from unseen distributions. While Vision Foundation Models (VFMs) like CLIP offer strong zero-shot capabilities, standard prompt tuning often causes models to overfit to the specific styles of the training domains. We introduce Disentangled Prompt Representation (DPR), a framework that leverages the semantic knowledge of Large Language Models (LLMs) to decouple object content from domain-specific styles. By using GPT-3 to generate distinct descriptions for invariant object features and domain-specific attributes, DPR guides the visual encoder to learn more robust representations. Our approach achieves new state-of-the-art results across five major DG benchmarks, including a significant improvement on the large-scale DomainNet dataset.

Introduction

A fundamental assumption in traditional machine learning is that training and testing data share the same distribution. However, real-world applications frequently encounter "domain shift"—for instance, a model trained on photographs of sheep may struggle to identify a sheep in a charcoal sketch or a stylized painting. This challenge, known as Domain Generalization (DG), requires models to distinguish between the essential features of an object (domain-invariant) and its stylistic presentation (domain-specific).

Conceptual overview of disentangling domain-invariant features (e.g., body shape) from domain-specific styles (e.g., color and line type) for a sheep.

Current methods using Vision Foundation Models like CLIP often rely on simple prompts (e.g., "a photo of a [class]"). While effective for zero-shot tasks, fine-tuning these prompts can lead the model to conflate object identity with training-specific styles, hindering generalization to new domains. The core difficulty lies in disentanglement: teaching a model to separate the "sheep-ness" from the "sketch-ness."

Methodology

The DPR framework addresses the disentanglement problem through a three-stage process:

1. GPT-Assisted Text-Disentanglement

The core innovation lies in using the semantic knowledge of LLMs to define the boundaries between content and style. Instead of generic prompts, we query GPT-3 to provide detailed descriptions of what an object looks like generally (domain-invariant) and how it appears in specific domains like "sketch" or "cartoon" (domain-specific). This creates a semantic roadmap for the vision model to follow.

2. Image-Disentanglement Guided by Text

Using these disentangled text descriptions as anchors, we employ Visual Prompt Tuning (VPT) to train two distinct visual pathways. One pathway is optimized to align with domain-invariant text, capturing the "essence" of the category. The second pathway aligns with domain-specific text, learning to represent the stylistic nuances of the training domains.

The DPR architecture: (a) Text-Disentanglement using GPT-3, (b) Image-Disentanglement via dual visual pathways, and (c) Domain-Specific Prototype Learning for inference.

3. Domain-Specific Prototype Learning (DSPL)

Rather than discarding style information during inference, DPR maintains a memory bank of prototypes from known domains. When presented with an image from an unseen domain, the model identifies the most similar known styles. It then combines a "style-aware" prediction with the "style-agnostic" prediction to produce a final classification, effectively leveraging its prior knowledge of different artistic or photographic styles.

Experimental Evaluation

We evaluated DPR on five standard DG benchmarks: PACS, VLCS, OfficeHome, DomainNet, and TerraInc. These datasets cover a wide range of domain shifts, from artistic styles to geographic variations in camera trap images.

Key Results

DPR established new state-of-the-art (SOTA) performance across all benchmarks. Notably:

  • DomainNet: Achieved 62.05% accuracy, a significant increase over the previous SOTA of 59.80%.
  • PACS: Reached 97.45% accuracy.
  • Data Efficiency: The framework performs exceptionally well in "few-shot" settings, requiring only 16 images per class for training.

Ablation studies confirmed that both the text-based disentanglement and the prototype-based inference are essential for these gains. Removing either component led to a measurable drop in generalization performance.

Conclusion

The Disentangled Prompt Representation (DPR) framework demonstrates that LLMs can serve as powerful semantic teachers for vision models. By explicitly "unmixing" content from style in the text modality, we can guide vision models toward more robust, domain-invariant representations. While the method introduces some computational overhead and a dependency on external LLMs, its superior performance on large-scale benchmarks like DomainNet highlights the potential of disentangled representations for solving the domain generalization challenge.

Summary updated

Abstract

Domain Generalization (DG) aims to develop a versatile model capable of performing well on unseen target domains. Recent advancements in pre-trained Visual Foundation Models (VFMs), such as CLIP, show significant potential in enhancing the generalization abilities of deep models. Although there is a growing focus on VFM-based domain prompt tuning for DG, effectively learning prompts that disentangle invariant features across all domains remains a major challenge. In this paper, we propose addressing this challenge by leveraging the controllable and flexible language prompt of the VFM. Observing that the text modality of VFMs is inherently easier to disentangle, we introduce a novel text feature guided visual prompt tuning framework. This framework first automatically disentangles the text prompt using a large language model (LLM) and then learns domain-invariant visual representation guided by the disentangled text feature. Moreover, we also devise domain-specific prototype learning to fully exploit domain-specific information to combine with the invariant feature prediction. Extensive experiments on mainstream DG datasets, namely PACS, VLCS, OfficeHome, DomainNet and TerraInc, demonstrate that the proposed method achieves superior performances to state-of-the-art DG methods.
BibTeX
@inproceedings{cheng2024disentangled,
  author = {De Cheng and Zhipeng Xu and Xinyang Jiang and Nannan Wang and Dongsheng Li and Xinbo Gao},
  title = {{Disentangled Prompt Representation for Domain Generalization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02227},
}
Figure