HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization

Guanglin Zhou, Zhongyi Han, Shiming Chen, Biwei Huang, Liming Zhu, Tongliang Liu, Lina Yao, Kun Zhang

TL;DR

Domain Generalization (DG) models often struggle to separate core, invariant features (like an object's shape) from domain-specific style variations (like texture or lighting). This paper introduces Hierarchical Contrastive Visual Prompt (HCVP), a novel method that generates instance-dependent, two-tier visual prompts (domain-level and task-specific) to explicitly guide a Vision Transformer (ViT) backbone. By separating domain noise from task features via prompt contrastive learning, HCVP achieves state-of-the-art generalization performance across multiple real-world benchmarks.

What Problem Does This Paper Address?

The holy grail of Domain Generalization (DG) is training a machine learning model on a set of source domains (e.g., photos, cartoons, sketches) so that it performs flawlessly on an entirely unseen target domain. To do this, a model must learn domain-invariant features (e.g., an elephant has large ears and a long nose) while ignoring domain-specific features (e.g., an elephant in a photo has thick grey skin, but in a sketch, it is just an outline).

Traditional DG methods apply uniform parameterization across the entire dataset, which inadvertently blends these two types of features together. While large visual foundation models like Vision Transformers (ViTs) offer a strong starting point, standard fine-tuning or prompt-tuning methods typically use static placeholders shared across all images. This lack of specificity makes it difficult for the model to adapt dynamically to nuanced inter-domain variations.

Fig. 1. Motivation illustration. (a) Traditional DG methods, employing univer- sal parameters across the entire dataset, often struggle to distinguish between invariant shape attributes (e.g., large ears, long nose) and domain-specific texture attributes (e.g., thick grey skin). This difficulty leads to a blending of features, thereby diminishing the model’s capacity for generalization. (b) Our approach introduces hierarchical visual prompts that encapsulate domain-level and task-specific characteristics, enabling the model to better differentiate and understand both invariant and specific attributes, thereby contributing to more effective generalization across different domains.

Methods

HCVP addresses this by introducing an explicit model structure and specialized loss functions designed to generate and inject instance-dependent visual prompts. The architecture consists of two primary components:

  1. Hierarchical Prompt Generation Network (HPGN): Instead of static prompt tokens, HPGN uses a pre-trained, frozen ResNet encoder to extract feature maps from each input image. It then generates prompts at two levels:
  2. Domain-level prompts: Capture coarse-grained, global features common across a domain using global average pooling (GAP) and a Multi-Layer Perceptron (MLP).
  3. Task-specific prompts: Capture fine-grained, localized patterns using convolutional layers that take both the image features and the domain-level prompt as inputs.
  4. Prompt Modulation Network (PMN): Acts as a bridge, mapping these dynamically generated prompts through an MLP structure that mirrors the ViT layers, systematically integrating them into each layer of the ViT backbone.
Architecture overview of the proposed HCVP model

To train this setup effectively, the authors introduce two contrastive learning strategies:

  • Prompt Contrastive Learning (PCL): Explicitly optimizes the prompts. Domain-level PCL forces prompts from the same domain to be similar and different domains to be distinct. Task-level PCL ensures prompts for the same class within a domain cluster together.
  • Class-conditioned Contrastive Invariance (CCI): Regularizes the final ViT embeddings, ensuring that representations of the same class remain close regardless of the domain they originated from, directly minimizing domain-specific influence.

Datasets

The authors evaluated HCVP on five standard DG benchmark datasets:

  • PACS: 9,991 images across 7 classes and 4 domains (Art Paintings, Cartoon, Photos, Sketches).
  • VLCS: 10,729 photographic images across 5 classes from 4 distinct datasets (Caltech-101, LabelMe, SUN09, VOC2007).
  • Office-Home: 15,588 images across 65 classes and 4 domains (Art, Clipart, Product, Real-World).
  • TerraIncognita: 24,788 wildlife images taken from 4 different camera trap locations.
  • CelebA: 27,040 images used to evaluate performance under spurious correlation shifts (classifying hair color against gender as a spurious attribute).

Key Insights and Ideas

  • Instance-Dependent Context Matters: Rather than relying on a single set of universal prompts for a dataset or domain, prompts should be dynamically generated per image. This provides the exact context needed to help the backbone isolate specific variations from invariant features.
  • Contrastive Learning on Prompts: Applying contrastive losses directly to the generative prompt space (via PCL) forces the prompt network to explicitly learn domain and task boundaries, rather than hoping the prompts implicitly pick up useful context during standard classification training.
  • Proven Domain Invariance: By measuring the average pairwise distance of latent features across domains, the authors verified that HCVP significantly reduces inter-domain feature distances compared to standard Empirical Risk Minimization (ERM). This confirms the model successfully extracts domain-invariant representations.
Comparison of inter-domain feature distances for ERM and HCVP

Key Results

HCVP was benchmarked against 18 established DG algorithms and 5 tuning protocols:

  • Top Overall Performance: HCVP achieved the highest overall average accuracy of 79.1% across all five datasets, outperforming strong baselines like SWAD, SAGM, and DoPrompt.
  • Robustness to Extreme Shifts: The advantage was most pronounced during severe distribution shifts. For instance, when generalizing to the Sketch domain in PACS, HCVP achieved 81.30% accuracy compared to ERM's 75.13%.
  • Efficiency Trade-off: Despite increasing total trainable parameters to ~90.1M (up from the baseline 85.8M due to the prompt generation networks), HCVP maintained a highly competitive GPU memory footprint (13.463 GB) and a fast training step time (1.249 seconds), vastly outperforming other adaptation protocols like the Transfer model (11.014 seconds).

Limitations and Future Work

While HCVP demonstrates remarkable robustness, the authors acknowledge that it may still fall short on exceptionally complex or highly out-of-distribution unseen domains. Furthermore, the current framework relies entirely on extracting domain and task characteristics from visual information alone. The authors suggest that incorporating cross-modal information (such as aligning visual features with textual domain descriptions) represents a promising future direction to further boost domain generalization capabilities.

Summary updated

Abstract

Domain Generalization (DG) endeavors to create machine learning models that excel in unseen scenarios by learning invariant features. In DG, the prevalent practice of constraining models to a fixed structure or uniform parameterization to encapsulate invariant features can inadvertently blend specific aspects. Such an approach struggles with nuanced differentiation of inter-domain variations and may exhibit bias towards certain domains, hindering the precise learning of domain-invariant features. Recognizing this, we introduce a novel method designed to supplement the model with domain-level and task-specific characteristics. This approach aims to guide the model in more effectively separating invariant features from specific characteristics, thereby boosting the generalization. Building on the emerging trend of visual prompts in the DG paradigm, our work introduces the novel \textbf{H}ierarchical \textbf{C}ontrastive \textbf{V}isual \textbf{P}rompt (HCVP) methodology. This represents a significant advancement in the field, setting itself apart with a unique generative approach to prompts, alongside an explicit model structure and specialized loss functions. Differing from traditional visual prompts that are often shared across entire datasets, HCVP utilizes a hierarchical prompt generation network enhanced by prompt contrastive learning. These generative prompts are instance-dependent, catering to the unique characteristics inherent to different domains and tasks. Additionally, we devise a prompt modulation network that serves as a bridge, effectively incorporating the generated visual prompts into the vision transformer backbone. Experiments conducted on five DG datasets demonstrate the effectiveness of HCVP, outperforming both established DG algorithms and adaptation protocols.
BibTeX
@article{zhou2025hcvp,
  author = {Guanglin Zhou and Zhongyi Han and Shiming Chen and Biwei Huang and Liming Zhu and Tongliang Liu and Lina Yao and Kun Zhang},
  title = {{HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=F8VqMbWgeK},
}
Figure