TL;DR
This paper introduces LMP, a novel dual-branch framework for Cross-Domain Few-Shot Object Detection (CD-FSOD). By combining the semantic understanding of text prompts with domain-specific visual prototypes, LMP significantly improves object detection accuracy in new domains with very limited labeled data, achieving state-of-the-art results.
What Problem Does This Paper Address?
The core challenge addressed is Cross-Domain Few-Shot Object Detection (CD-FSOD). Traditional Few-Shot Object Detection (FSOD) assumes that the training and testing data come from the same domain. However, in real-world scenarios, domain shifts are common (e.g., detecting objects in aerial images versus underwater images). Existing methods that leverage powerful Vision-Language Models (VLMs) like CLIP and GroundingDINO excel at open-vocabulary detection by using text prompts. While these prompts capture high-level semantics, they fail to account for the visual appearance of objects in a new, unseen domain. This is particularly problematic in few-shot settings where there's very little data to learn these domain-specific visual cues. The paper highlights that text prompts alone miss crucial visual information like texture, lighting, and background context, leading to poor localization when domains differ. Furthermore, with limited examples, models struggle to distinguish true objects from visually similar background distractors, leading to false positives.
Methods
The proposed solution is a dual-branch framework called LMP (Learning Multi-Modal Prototypes). It builds upon the existing GroundingDINO architecture, preserving its text-guided branch for open-vocabulary semantics. The key innovation is the introduction of a parallel visual-guided branch.
Here's how it works:
- Visual Prototype Construction: This module is central to the visual branch.
- Class-level Prototypes: These are generated by aggregating features from Region of Interest (RoI) proposals within the limited "support" images of the novel classes. This captures the typical visual appearance of each class in the target domain.
- Hard Negative Prototypes: To combat false positives from visually similar backgrounds or distractors, these are dynamically generated. For each ground-truth bounding box in a "query" image, the system samples perturbed boxes (using random jittering) that have a low IoU with the ground truth but still overlap enough to be potentially confusing. These perturbed regions are then used to create negative prototypes.
- Dual-Branch Architecture:
- Text-Guided Branch: This branch functions similarly to standard VLM-based detectors, using text prompts for semantic guidance.
- Visual-Guided Branch: This branch injects the constructed visual prototypes (both class-level and hard negative) into the detection pipeline. It uses these prototypes to guide feature enhancement, query selection, and the final decoding process.
- Joint Training and Ensemble: Both branches are trained jointly. At inference time, their predictions are ensembled to combine the semantic understanding from the text branch with the domain-adaptive visual details from the visual branch. This synergy allows for more robust and accurate detection.
Datasets
The paper evaluates its method on six cross-domain benchmark datasets:
- ArTaxOr: For photorealistic images.
- Clipart1k: For cartoon illustrations.
- DIOR: For aerial photography.
- DeepFish: For underwater imagery.
- NEU-DET: For industrial defect inspection.
- UODD: For underwater imagery.
The experiments are conducted under standard 1-shot, 5-shot, and 10-shot settings. The paper also mentions using the NWPU VHR-10 remote sensing dataset for additional evaluation.
Key Insights and Ideas
- Visual Prototypes are Crucial for Domain Adaptation: The core insight is that while text prompts provide semantic understanding, they are insufficient for adapting to new visual domains. Explicitly learning visual prototypes from the target domain's limited data is essential for bridging the domain gap.
- Hard Negatives are Key to Robustness: In few-shot scenarios, distinguishing objects from visually similar backgrounds is a major hurdle. Dynamically generating "hard negative" prototypes from jittered bounding boxes effectively teaches the model to suppress false positives, leading to significant performance gains, especially in low-shot settings.
- Dual-Branch Synergy: Combining a text-guided branch (for general semantics) with a visual-guided branch (for domain-specific appearance) offers a powerful synergistic effect. This allows the model to leverage the strengths of both modalities, leading to state-of-the-art performance.
- Prototype-Guided Attention: The visual prototypes are not just static features; they actively guide the attention mechanisms within the detector, influencing feature enhancement, query selection, and the final output.
Key Results
- State-of-the-Art Performance: LMP achieves state-of-the-art or highly competitive mean Average Precision (mAP) across all tested cross-domain benchmarks and shot settings (1, 5, and 10-shot).
- Significant Gains in 1-Shot: The method shows particularly strong improvements in the 1-shot regime, demonstrating its effectiveness under extreme data scarcity. For instance, it improves upon the GroundingDINO baseline by 8.0 mAP in the 1-shot setting.
- Effectiveness on Coarse-Label Datasets: LMP excels on datasets with coarse taxonomic labels (like "Coleoptera") where text alone provides weak guidance, by injecting domain-specific visual cues.
- Ablation Studies Confirm Contributions: Ablation studies show that adding class-level prototypes consistently improves performance over text-only baselines, and further incorporating hard negative prototypes leads to optimal results.
- Optimal Hyperparameter Settings: The study found that using three hard negative prototypes per ground truth and an equal weighting factor (α=1.0) for the visual and text branches yields the best performance.
Limitations and Future Work
- Limitations:
- Sensitivity to Non-Typical Supports: The performance might be affected if the few support images are not representative of the target domain.
- Computational Overhead: Running a dual-branch architecture naturally incurs higher computational costs compared to a single-branch model.
- Future Work:
- Adaptive Prototype Creation and Pruning: Developing methods to automatically select and refine prototypes.
- Stronger Negative Mining: Exploring more advanced techniques for identifying and utilizing negative examples (e.g., using ring/context regions or proposal similarity).
- Lightweight Support Augmentation and Distillation: Investigating ways to reduce the computational burden and distill the knowledge into a more efficient model.
- Extending to Richer Prompts and Modalities: Exploring the use of richer textual prompts, semi-supervised support sets, and even video or multi-view settings.
Summary updated