Abstract
Few-shot object detection (FSOD) aims to detect objects with only a few training examples. Visual feature extraction and query-support similarity learning are the two critical components. Existing works are usually developed based on ImageNet pre-trained vision backbones and design sophisticated metric-learning networks for few-shot learning, but still have inferior accuracy. In this work, we study few-shot object detection using modern foundation models. First, vision-only contrastive pre-trained DINOv2 model is used for the vision backbone, which shows strong transferable performance without tuning the parameters. Second, Large Language Model (LLM) is employed for contextualized few-shot learning with the input of all classes and query image proposals. Language instructions are carefully designed to prompt the LLM to classify each proposal in context. The contextual information include proposal-proposal relations, proposal-class relations, and class-class relations, which can largely promote few-shot learning. We comprehensively evaluate the proposed model (FM-FSOD) in multiple FSOD benchmarks, achieving state-of-the-arts performance.
Introduction
Few-Shot Object Detection (FSOD) is the challenge of teaching a model to detect new objects using only a handful of examples (e.g., 1, 5, or 10 images). Historically, researchers have relied on backbones pre-trained on ImageNet and spent significant effort designing complex "metric-learning" networks to compare query images with support (example) images.
Despite these efforts, FSOD performance has lagged behind standard object detection. The authors of FM-FSOD argue that the bottleneck lies in the insufficient utilization of modern foundation models. Existing FSOD models often require extensive fine-tuning, which can lead to "forgetting" general features, or they use simple similarity measures that fail to capture the complex relationships between different objects in a scene.
Methods
The proposed FM-FSOD framework leverages the power of foundation models through three main submodules:
- Visual Feature Extraction (DINOv2): Instead of fine-tuning a backbone, the authors use a frozen DINOv2 model. Trained via self-supervision on massive datasets, DINOv2 provides highly discriminative, spatially aware features that generalize well to novel classes without needing parameter updates.
- Proposal Generation: The model uses a Transformer-based architecture to generate potential object locations. It applies cross-attention between the query image and "class prototypes" (averaged features of the few-shot examples) to ensure the proposals are relevant to the target classes.
- Few-Shot Classification via LLM: This is the core innovation. The authors treat classification as a language task. Visual features are mapped into the LLM's word embedding space, and a Large Language Model (Vicuna) is prompted to classify the proposals. The prompt includes a mapping table of all classes and their visual prototypes, enabling the LLM to use in-context learning to distinguish between classes and background.
Key Insights
- Frozen Backbones for Generalization: Freezing the DINOv2 backbone performs better than fine-tuning it. Fine-tuning on limited data can degrade the rich, general-purpose features learned during pre-training.
- Contextualized Classification: Traditional FSOD models classify each proposal in isolation. By using an LLM, FM-FSOD performs contextualized classification, looking at all proposals and classes simultaneously to exploit their mutual relations.
- LLMs as Visual Reasoners: The paper demonstrates that LLMs can effectively act as the "head" of a vision model, leveraging internal logic to handle the noisy similarity matching typical of few-shot learning.
Results
FM-FSOD was evaluated on the MSCOCO and PASCAL VOC benchmarks, achieving significant performance gains over previous state-of-the-art (SOTA) methods. On MSCOCO 30-shot, it achieved 37.0 AP, outperforming the previous best ViT-based model (DE-ViT) by 3.0 points. It consistently achieved SOTA results across all PASCAL VOC splits, particularly in higher-shot scenarios.
Limitations and Future Work
The primary drawback is computational efficiency. The LLM-based classification head is significantly slower than standard heads, taking roughly 20 hours to evaluate 5,000 images compared to 25 minutes for standard models. Future work will focus on efficient LLM inference for vision tasks and improving handling of extremely small or occluded objects.
Summary updated