ProD: Prompting-to-disentangle Domain Knowledge for Cross-domain Few-shot Image Classification

Tianyi Ma, Yifan Sun, Zongxin Yang, Yi YangCVPR 2023 · OpenReview

Abstract

This paper considers few-shot image classification under the cross-domain scenario, where the train-to-test domain gap compromises classification accuracy. To mitigate the domain gap, we propose a prompting-to-disentangle (ProD) method through a novel exploration with the prompting mechanism. ProD adopts the popular multi-domain training scheme and extracts the backbone feature with a standard Convolutional Neural Network. Based on these two common practices, the key point of ProD is using the prompting mechanism in the transformer to disentangle the domain-general (DG) and domain-specific (DS) knowledge from the backbone feature. Specifically, ProD concatenates a DG and a DS prompt to the backbone feature and feeds them into a lightweight transformer. The DG prompt is learnable and shared by all the training domains, while the DS prompt is generated from the domain-of-interest on the fly. As a result, the transformer outputs DG and DS features in parallel with the two prompts, yielding the disentangling effect. We show that: 1) Simply sharing a single DG prompt for all the training domains already improves generalization towards the novel test domain. 2) The cross-domain generalization can be further reinforced by making the DG prompt neutral towards the training domains. 3) When inference, the DS prompt is generated from the support samples and can capture test domain knowledge through the prompting mechanism. Combining all three benefits, ProD significantly improves cross-domain few-shot classification. For instance, on CUB, ProD improves the 5-way 5-shot accuracy from 73.56% (baseline) to 79.19%, setting a new state of the art.

Introduction

Few-shot image classification aims to recognize new categories using only a handful of examples. However, a major hurdle in real-world applications is the cross-domain scenario: the training data (e.g., general images from ImageNet) looks very different from the test data (e.g., specific bird species or satellite imagery). Standard models often "overfit" to the training domain's style, making their features less useful for the target domain. Existing solutions usually try to either ignore domain-specific details entirely (Domain Generalization) or fine-tune the model on the few available target samples (Domain Adaptation). Fine-tuning is risky with only 1–5 images, as it often leads to catastrophic forgetting or poor optimization. ProD seeks a middle ground: a way to capture the "essence" of the new domain dynamically without changing the model's weights.

Methodology: Prompting-to-disentangle (ProD)

The authors propose ProD (Prompting-to-disentangle), which sits on top of a standard CNN backbone (like ResNet-10) and a lightweight Transformer. The core innovation lies in how they use prompts—extra learnable tokens—to guide the Transformer.

Overview of the ProD architecture showing the disentanglement of DG and DS knowledge via prompts.

1. Domain-General (DG) Prompt

This is a set of learnable tokens shared across all training domains. It is designed to capture universal features. To ensure it doesn't become biased, the authors use a Neutralization Loss, which forces the DG prompt to stay "equidistant" from the feature centers of all training domains.

2. Domain-Specific (DS) Prompt

Unlike the DG prompt, this isn't fixed. It is generated on-the-fly by averaging the backbone features of the current domain. During testing, the DS prompt is generated directly from the few "support" images provided for the new task.

Detailed ProD pipeline (a) and the mechanism for neutralizing the DG prompt (b).

3. Parallel Disentanglement and Inference

The Transformer processes the image features along with both prompts simultaneously. The output associated with the DG prompt is sent to a global classifier (covering all training classes), while the output associated with the DS prompt is sent to a local classifier (covering only classes within that specific domain). At test time, the model concatenates the DG and DS features to create a robust final representation that benefits from both general knowledge and immediate context from the new domain.

Key Results and Insights

ProD sets a new state-of-the-art in cross-domain few-shot classification. On the CUB dataset (5-way 5-shot), ProD achieved 79.19% accuracy, a significant jump from the 73.56% baseline. Across all target domains (CUB, Cars, Plantae, Places), ProD consistently outperformed competitors.

  • Prompts as "Switches": Prompts act as a mechanism to "switch" the Transformer's attention between general structural features and domain-specific textures or styles.
  • DS Knowledge is an Asset: ProD argues that DS knowledge is actually highly valuable if it can be extracted from the target domain at test time.
  • Neutrality Breeds Generalization: By explicitly penalizing the DG prompt for being too close to any one training domain, the model is forced to learn features that are truly domain-agnostic.
  • Efficiency: ProD adds negligible parameter overhead (~0.1M parameters) compared to the baseline "CNN + Transformer" setup, yet yields much higher gains.

Conclusion

ProD introduces a novel prompting mechanism that disentangles domain-general knowledge from domain-specific knowledge to tackle the "domain gap" in few-shot learning. By using a shared "neutral" prompt for general features and an on-the-fly generated prompt for domain-specific features, it achieves state-of-the-art accuracy on cross-domain benchmarks without requiring expensive test-time fine-tuning.

Summary updated

Abstract

This paper considers few-shot image classification under the cross-domain scenario, where the train-to-test domain gap compromises classification accuracy. To mitigate the domain gap, we propose a prompting-to-disentangle (ProD) method through a novel exploration with the prompting mechanism. ProD adopts the popular multi-domain training scheme and extracts the backbone feature with a standard Convolutional Neural Network. Based on these two common practices, the key point of ProD is using the prompting mechanism in the transformer to disentangle the domain-general (DG) and domain-specific (DS) knowledge from the backbone feature. Specifically, ProD concatenates a DG and a DS prompt to the backbone feature and feeds them into a lightweight transformer. The DG prompt is learnable and shared by all the training domains, while the DS prompt is generated from the domain-of-interest on the fly. As a result, the transformer outputs DG and DS features in parallel with the two prompts, yielding the disentangling effect. We show that: 1) Simply sharing a single DG prompt for all the training domains already improves generalization towards the novel test domain. 2) The cross-domain generalization can be further reinforced by making the DG prompt neutral towards the training domains. 3) When inference, the DS prompt is generated from the support samples and can capture test domain knowledge through the prompting mechanism. Combining all three benefits, ProD significantly improves cross-domain few-shot classification. For instance, on CUB, ProD improves the 5-way 5-shot ac-curacy from 73.56% (baseline) to 79.19%, setting a new state of the art.
BibTeX
@inproceedings{ma2023prod,
  author = {Tianyi Ma and Yifan Sun and Zongxin Yang and Yi Yang},
  title = {{ProD: Prompting-to-disentangle Domain Knowledge for Cross-domain Few-shot Image Classification}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=XFW9qZf4gB},
}
Figure