TL;DR
This paper introduces VisTex-OVLM, a novel method for few-shot object detection that uses "visual textualization" to convert image exemplars into text-like features. This allows existing vision-language models (OVLMs) to leverage visual prompts without disrupting their pre-trained object-text alignment, achieving state-of-the-art results on challenging open-set and few-shot detection benchmarks.
What Problem Does This Paper Address?
Modern vision-language models (VLMs), particularly object-level VLMs (OVLMs) like GLIP, excel at zero-shot object detection by leveraging extensive pre-training on image-text data. However, they struggle with detecting rare categories that are underrepresented in their training data. While few-shot object detection (FSOD) methods can address this by using a few annotated examples (exemplars), directly fine-tuning VLMs on these limited examples can degrade their general object-text alignment and overall generalization capabilities. Existing methods that try to incorporate visual prompts often modify the OVLM architecture, risking this crucial alignment. The core problem is how to effectively use visual exemplars to improve detection of novel categories without compromising the robust, pre-trained capabilities of OVLMs.
Methods
The proposed method, VisTex-OVLM, tackles this by introducing visual textualization. Instead of fine-tuning the OVLM, it converts the visual information from a few "support" images (exemplars) into a textualized format that can be directly fed into the OVLM's existing text prompt input.
Here's a breakdown of the key components:
- Image Prompt Engineering: Support images are first processed to highlight the object of interest, for example, by applying background blur to regions outside the target bounding box.
- Multi-Scale Textualizing Blocks (MSTBs): These are lightweight, parameter-shared modules designed to process multi-scale visual features extracted from the OVLM's own visual encoder. MSTBs project these visual features into the OVLM's text feature space. The parameter sharing across scales is crucial for efficient training and knowledge transfer.
- Multi-Stage Fusion (MSF): This strategy integrates the textualized visual features from different stages of the OVLM's visual encoder. It's a non-parametric fusion that leverages the OVLM's multi-stage architecture without adding extra trainable parameters.
- Direct Support Image Prompting: The output of the MSTB and MSF is a single "textualized visual token" for each support image. This token is then concatenated with the standard text prompt (e.g., class name) and fed into the unmodified, pre-trained OVLM. This direct integration preserves the OVLM's original architecture and object-text alignment.
Datasets
The paper evaluates VisTex-OVLM on several datasets:
- Open-set Scenarios:
- LVIS MiniVal: A challenging dataset known for its long-tail distribution of object categories.
- 16 Unseen Datasets: A collection of datasets with minimal overlap with the OVLM's pre-training data, including medical datasets and general object detection datasets (ODinW35).
- Standard Few-Shot Object Detection (FSOD) Benchmarks:
- PASCAL VOC: Divided into three splits, each with 15 base and 5 novel categories.
- MSCOCO: Uses 20 categories overlapping with PASCAL VOC as novel classes and the remaining 60 as base classes.
Key Insights and Ideas
- Visual Information as Text: The core idea of "visual textualization" is powerful. By transforming visual exemplars into a format that the text encoder can understand, the method bridges the gap between visual and textual modalities in a way that respects the OVLM's existing architecture.
- Preserving Object-Text Alignment is Crucial: The paper strongly emphasizes that fine-tuning or adding complex modules can break the delicate balance of object-text alignment in OVLMs. VisTex-OVLM's success hinges on its ability to avoid this, leading to better generalization.
- Leveraging Existing OVLM Strengths: Instead of reinventing the wheel, VisTex-OVLM cleverly utilizes the multi-scale and multi-stage nature of existing OVLMs, enhancing their capabilities without altering their fundamental design.
- Text Prompts Still Matter: Attention heatmaps reveal that while visual prompts are powerful, text prompts are still essential for providing semantic guidance and maintaining strong image-text alignment.
Key Results
VisTex-OVLM achieves state-of-the-art (SOTA) performance across various challenging scenarios:
- Open-set Detection: It significantly outperforms existing methods on LVIS MiniVal and the 16 unseen datasets, demonstrating its effectiveness in detecting categories not seen during pre-training.
- Few-Shot Detection: It achieves SOTA results on standard FSOD benchmarks like PASCAL VOC and MSCOCO, outperforming both non-VLM-based and other VLM-based few-shot methods.
- Generalization: The method shows strong performance on generalized FSOD (GFSOD) tasks, maintaining high accuracy on base classes while detecting novel ones, indicating it avoids overfitting and knowledge forgetting.
- Efficiency: The MSTBs are designed to be lightweight, and the MSF strategy is non-parametric, contributing to computational efficiency.
Limitations and Future Work
- Text Prompts Remain Important: The paper acknowledges that while visual prompts are beneficial, they are most effective when combined with text prompts. Purely visual prompting without text can lead to weaker performance. This suggests future work could explore more sophisticated ways to integrate visual and textual cues.
- Computational Overhead: While the method aims to be efficient, the introduction of MSTBs does add some trainable parameters (though significantly fewer than full fine-tuning). The paper mentions ablation studies on computational overhead in the supplementary materials.
- Applicability to Other OVLMs: The paper demonstrates VisTex-OVLM's effectiveness on GLIP and GroundingDINO, suggesting broad applicability. Future work could explore its integration with an even wider range of OVLMs.
- Further Exploration of Prompt Engineering: The paper mentions image prompt engineering (e.g., background blur) as a step. Further research into more advanced prompt engineering techniques for visual exemplars could yield even better results.
Summary updated