TL;DR
Domain Generalization (DG) models often struggle to separate core, invariant features (like an object's shape) from domain-specific style variations (like texture or lighting). This paper introduces Hierarchical Contrastive Visual Prompt (HCVP), a novel method that generates instance-dependent, two-tier visual prompts (domain-level and task-specific) to explicitly guide a Vision Transformer (ViT) backbone. By separating domain noise from task features via prompt contrastive learning, HCVP achieves state-of-the-art generalization performance across multiple real-world benchmarks.
What Problem Does This Paper Address?
The holy grail of Domain Generalization (DG) is training a machine learning model on a set of source domains (e.g., photos, cartoons, sketches) so that it performs flawlessly on an entirely unseen target domain. To do this, a model must learn domain-invariant features (e.g., an elephant has large ears and a long nose) while ignoring domain-specific features (e.g., an elephant in a photo has thick grey skin, but in a sketch, it is just an outline).
Traditional DG methods apply uniform parameterization across the entire dataset, which inadvertently blends these two types of features together. While large visual foundation models like Vision Transformers (ViTs) offer a strong starting point, standard fine-tuning or prompt-tuning methods typically use static placeholders shared across all images. This lack of specificity makes it difficult for the model to adapt dynamically to nuanced inter-domain variations.
Methods
HCVP addresses this by introducing an explicit model structure and specialized loss functions designed to generate and inject instance-dependent visual prompts. The architecture consists of two primary components:
- Hierarchical Prompt Generation Network (HPGN): Instead of static prompt tokens, HPGN uses a pre-trained, frozen ResNet encoder to extract feature maps from each input image. It then generates prompts at two levels:
- Domain-level prompts: Capture coarse-grained, global features common across a domain using global average pooling (GAP) and a Multi-Layer Perceptron (MLP).
- Task-specific prompts: Capture fine-grained, localized patterns using convolutional layers that take both the image features and the domain-level prompt as inputs.
- Prompt Modulation Network (PMN): Acts as a bridge, mapping these dynamically generated prompts through an MLP structure that mirrors the ViT layers, systematically integrating them into each layer of the ViT backbone.
To train this setup effectively, the authors introduce two contrastive learning strategies:
- Prompt Contrastive Learning (PCL): Explicitly optimizes the prompts. Domain-level PCL forces prompts from the same domain to be similar and different domains to be distinct. Task-level PCL ensures prompts for the same class within a domain cluster together.
- Class-conditioned Contrastive Invariance (CCI): Regularizes the final ViT embeddings, ensuring that representations of the same class remain close regardless of the domain they originated from, directly minimizing domain-specific influence.
Datasets
The authors evaluated HCVP on five standard DG benchmark datasets:
- PACS: 9,991 images across 7 classes and 4 domains (Art Paintings, Cartoon, Photos, Sketches).
- VLCS: 10,729 photographic images across 5 classes from 4 distinct datasets (Caltech-101, LabelMe, SUN09, VOC2007).
- Office-Home: 15,588 images across 65 classes and 4 domains (Art, Clipart, Product, Real-World).
- TerraIncognita: 24,788 wildlife images taken from 4 different camera trap locations.
- CelebA: 27,040 images used to evaluate performance under spurious correlation shifts (classifying hair color against gender as a spurious attribute).
Key Insights and Ideas
- Instance-Dependent Context Matters: Rather than relying on a single set of universal prompts for a dataset or domain, prompts should be dynamically generated per image. This provides the exact context needed to help the backbone isolate specific variations from invariant features.
- Contrastive Learning on Prompts: Applying contrastive losses directly to the generative prompt space (via PCL) forces the prompt network to explicitly learn domain and task boundaries, rather than hoping the prompts implicitly pick up useful context during standard classification training.
- Proven Domain Invariance: By measuring the average pairwise distance of latent features across domains, the authors verified that HCVP significantly reduces inter-domain feature distances compared to standard Empirical Risk Minimization (ERM). This confirms the model successfully extracts domain-invariant representations.
Key Results
HCVP was benchmarked against 18 established DG algorithms and 5 tuning protocols:
- Top Overall Performance: HCVP achieved the highest overall average accuracy of 79.1% across all five datasets, outperforming strong baselines like SWAD, SAGM, and DoPrompt.
- Robustness to Extreme Shifts: The advantage was most pronounced during severe distribution shifts. For instance, when generalizing to the Sketch domain in PACS, HCVP achieved 81.30% accuracy compared to ERM's 75.13%.
- Efficiency Trade-off: Despite increasing total trainable parameters to ~90.1M (up from the baseline 85.8M due to the prompt generation networks), HCVP maintained a highly competitive GPU memory footprint (13.463 GB) and a fast training step time (1.249 seconds), vastly outperforming other adaptation protocols like the Transfer model (11.014 seconds).
Limitations and Future Work
While HCVP demonstrates remarkable robustness, the authors acknowledge that it may still fall short on exceptionally complex or highly out-of-distribution unseen domains. Furthermore, the current framework relies entirely on extracting domain and task characteristics from visual information alone. The authors suggest that incorporating cross-modal information (such as aligning visual features with textual domain descriptions) represents a promising future direction to further boost domain generalization capabilities.
Summary updated