Learning Multi-Modal Prototypes for Cross-Domain Few-Shot Object Detection

Wanqi Wang, Jingcai Guo, Yuxiang Cai, Zhi ChenarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces LMP, a novel dual-branch framework for Cross-Domain Few-Shot Object Detection (CD-FSOD). By combining the semantic understanding of text prompts with domain-specific visual prototypes, LMP significantly improves object detection accuracy in new domains with very limited labeled data, achieving state-of-the-art results.

What Problem Does This Paper Address?

The core challenge addressed is Cross-Domain Few-Shot Object Detection (CD-FSOD). Traditional Few-Shot Object Detection (FSOD) assumes that the training and testing data come from the same domain. However, in real-world scenarios, domain shifts are common (e.g., detecting objects in aerial images versus underwater images). Existing methods that leverage powerful Vision-Language Models (VLMs) like CLIP and GroundingDINO excel at open-vocabulary detection by using text prompts. While these prompts capture high-level semantics, they fail to account for the visual appearance of objects in a new, unseen domain. This is particularly problematic in few-shot settings where there's very little data to learn these domain-specific visual cues. The paper highlights that text prompts alone miss crucial visual information like texture, lighting, and background context, leading to poor localization when domains differ. Furthermore, with limited examples, models struggle to distinguish true objects from visually similar background distractors, leading to false positives.

Methods

The proposed solution is a dual-branch framework called LMP (Learning Multi-Modal Prototypes). It builds upon the existing GroundingDINO architecture, preserving its text-guided branch for open-vocabulary semantics. The key innovation is the introduction of a parallel visual-guided branch.

Here's how it works:

  • Visual Prototype Construction: This module is central to the visual branch.
  • Class-level Prototypes: These are generated by aggregating features from Region of Interest (RoI) proposals within the limited "support" images of the novel classes. This captures the typical visual appearance of each class in the target domain.
  • Hard Negative Prototypes: To combat false positives from visually similar backgrounds or distractors, these are dynamically generated. For each ground-truth bounding box in a "query" image, the system samples perturbed boxes (using random jittering) that have a low IoU with the ground truth but still overlap enough to be potentially confusing. These perturbed regions are then used to create negative prototypes.
  • Dual-Branch Architecture:
    • Text-Guided Branch: This branch functions similarly to standard VLM-based detectors, using text prompts for semantic guidance.
  • Visual-Guided Branch: This branch injects the constructed visual prototypes (both class-level and hard negative) into the detection pipeline. It uses these prototypes to guide feature enhancement, query selection, and the final decoding process.
  • Joint Training and Ensemble: Both branches are trained jointly. At inference time, their predictions are ensembled to combine the semantic understanding from the text branch with the domain-adaptive visual details from the visual branch. This synergy allows for more robust and accurate detection.

Datasets

The paper evaluates its method on six cross-domain benchmark datasets:

  • ArTaxOr: For photorealistic images.
  • Clipart1k: For cartoon illustrations.
  • DIOR: For aerial photography.
  • DeepFish: For underwater imagery.
  • NEU-DET: For industrial defect inspection.
  • UODD: For underwater imagery.

The experiments are conducted under standard 1-shot, 5-shot, and 10-shot settings. The paper also mentions using the NWPU VHR-10 remote sensing dataset for additional evaluation.

Key Insights and Ideas

  • Visual Prototypes are Crucial for Domain Adaptation: The core insight is that while text prompts provide semantic understanding, they are insufficient for adapting to new visual domains. Explicitly learning visual prototypes from the target domain's limited data is essential for bridging the domain gap.
  • Hard Negatives are Key to Robustness: In few-shot scenarios, distinguishing objects from visually similar backgrounds is a major hurdle. Dynamically generating "hard negative" prototypes from jittered bounding boxes effectively teaches the model to suppress false positives, leading to significant performance gains, especially in low-shot settings.
  • Dual-Branch Synergy: Combining a text-guided branch (for general semantics) with a visual-guided branch (for domain-specific appearance) offers a powerful synergistic effect. This allows the model to leverage the strengths of both modalities, leading to state-of-the-art performance.
  • Prototype-Guided Attention: The visual prototypes are not just static features; they actively guide the attention mechanisms within the detector, influencing feature enhancement, query selection, and the final output.

Key Results

  • State-of-the-Art Performance: LMP achieves state-of-the-art or highly competitive mean Average Precision (mAP) across all tested cross-domain benchmarks and shot settings (1, 5, and 10-shot).
  • Significant Gains in 1-Shot: The method shows particularly strong improvements in the 1-shot regime, demonstrating its effectiveness under extreme data scarcity. For instance, it improves upon the GroundingDINO baseline by 8.0 mAP in the 1-shot setting.
  • Effectiveness on Coarse-Label Datasets: LMP excels on datasets with coarse taxonomic labels (like "Coleoptera") where text alone provides weak guidance, by injecting domain-specific visual cues.
  • Ablation Studies Confirm Contributions: Ablation studies show that adding class-level prototypes consistently improves performance over text-only baselines, and further incorporating hard negative prototypes leads to optimal results.
  • Optimal Hyperparameter Settings: The study found that using three hard negative prototypes per ground truth and an equal weighting factor (α=1.0) for the visual and text branches yields the best performance.

Limitations and Future Work

  • Limitations:
    • Sensitivity to Non-Typical Supports: The performance might be affected if the few support images are not representative of the target domain.
  • Computational Overhead: Running a dual-branch architecture naturally incurs higher computational costs compared to a single-branch model.
  • Future Work:
    • Adaptive Prototype Creation and Pruning: Developing methods to automatically select and refine prototypes.
  • Stronger Negative Mining: Exploring more advanced techniques for identifying and utilizing negative examples (e.g., using ring/context regions or proposal similarity).
  • Lightweight Support Augmentation and Distillation: Investigating ways to reduce the computational burden and distill the knowledge into a more efficient model.
  • Extending to Richer Prompts and Modalities: Exploring the use of richer textual prompts, semi-supervised support sets, and even video or multi-view settings.

Summary updated

Abstract

Cross-Domain Few-Shot Object Detection (CD-FSOD) aims to detect novel classes in unseen target domains given only a few labeled examples. While open-vocabulary detectors built on vision-language models (VLMs) transfer well, they depend almost entirely on text prompts, which encode domain-invariant semantics but miss domain-specific visual information needed for precise localization under few-shot supervision. We propose a dual-branch detector that Learns Multi-modal Prototypes, dubbed LMP, by coupling textual guidance with visual exemplars drawn from the target domain. A Visual Prototype Construction module aggregates class-level prototypes from support RoIs and dynamically generates hard-negative prototypes in query images via jittered boxes, capturing distractors and visually similar backgrounds. In the visual-guided branch, we inject these prototypes into the detection pipeline with components mirrored from the text branch as the starting point for training, while a parallel text-guided branch preserves open-vocabulary semantics. The branches are trained jointly and ensembled at inference by combining semantic abstraction with domain-adaptive details. On six cross-domain benchmark datasets and standard 1/5/10-shot settings, our method achieves state-of-the-art or highly competitive mAP.
BibTeX
@misc{wang2026learning,
  author = {Wanqi Wang and Jingcai Guo and Yuxiang Cai and Zhi Chen},
  title = {{Learning Multi-Modal Prototypes for Cross-Domain Few-Shot Object Detection}},
  year = {2026},
  eprint = {2602.18811},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.18811},
}

Loading related content...

Figure