Abstract
Few-shot learning has recently emerged as a new challenge in the deep learning field: unlike conventional methods that train deep neural networks (DNNs) with a large number of labeled data, it asks for the generalization of DNNs on new classes with few annotated samples. Recent advances in few-shot learning mainly focus on image classification, while in this paper we focus on object detection. The initial explorations in few-shot object detection tend to simulate a classification scenario by using the positive proposals in images with respect to certain object class while discarding the negative proposals of that class. Negatives, especially hard negatives, however, are essential to the embedding space learning in few-shot object detection. In this paper, we restore the negative information in few-shot object detection by introducing a new negative- and positive-representative based metric learning framework and a new inference scheme with negative and positive representatives. We build our work on a recent few-shot pipeline RepMet with several new modules to encode negative information for both training and testing. Extensive experiments on ImageNet-LOC and PASCAL VOC show our method substantially improves the state-of-the-art few-shot object detection solutions.
Introduction
Object detection is a fundamental task in computer vision, yet "few-shot" object detection (FSOD) remains a significant challenge. While humans can generalize from a single example—identifying a rare animal in a dense forest after seeing just one photograph—deep neural networks typically require thousands of annotated samples to achieve similar performance.
Current FSOD methods often borrow heavily from few-shot classification paradigms. They typically look at a "support" image (the example), extract the object of interest, and attempt to find similar patches in a "query" image. However, a major oversight in these models is the exclusion of "negative proposals." These are patches of an image that contain only fragments of an object, ambiguous background clutter, or shapes that resemble the target but are not it. By ignoring these "hard negatives," models struggle to define the precise boundaries of an object's feature space, leading to high false-positive rates. This research proposes NP-RepMet, a framework designed to restore this "trash" data to build a more robust detector.
The NP-RepMet Framework
Building upon the existing metric-learning-based detector RepMet, the authors introduce several key components to integrate negative information:
- Dual Representatives: Instead of learning only a "positive" prototype for a class (e.g., what a "car" looks like), the model learns both positive and negative representatives. This allows the model to explicitly learn the characteristics of "not-quite-a-car."
- NP-Embedding: The network produces two distinct feature vectors for every image proposal: a positive embedding ($E^p$) and a negative embedding ($E^n$). This dual-embedding approach allows the model to converge more accurately by separating features that define the object from those that define its common distractors.
- Triplet Loss Steering: A specialized triplet loss function is employed. For a given proposal, the loss pushes its embedding toward the corresponding positive prototypes and away from the negative ones.
Key Insights: The Role of Hard Negatives
A central finding of this work is that "hard negatives"—patches with low overlap (IoU 0.2–0.3) with the ground truth—are more valuable than pure background. These "almost-objects" provide the necessary contrast to prevent the model from being confused by partial views or similar shapes.
To manage the vast amount of negative data, the authors utilize spectral clustering during inference. This ensures the model selects a diverse set of representative hard negatives from the support set, rather than focusing on a single type of background clutter. This diversity is crucial for structuring the embedding space such that "positive zones" become distinct and well-defined.
Experimental Results
NP-RepMet was evaluated on two major benchmarks: ImageNet-LOC (focusing on 214 unseen animal classes) and PASCAL VOC 2007 (using standard FSOD splits). The method consistently outperformed state-of-the-art (SOTA) solutions, particularly in data-scarce scenarios:
- ImageNet-LOC: Achieved an 11.6% absolute improvement in 1-shot mAP (from 56.9% to 68.5%).
- PASCAL VOC: On the second novel set split, 1-shot performance saw a massive 19.8% gain, jumping from 21.8% to 41.6%.
The results indicate that the relative gain of restoring negative information is highest when training data is most limited, though improvements were consistent across 1, 2, 3, 5, and 10-shot scenarios.
Conclusion and Future Work
Restoring negative information fundamentally changes the structure of the embedding space in few-shot detection. While highly effective, the approach does introduce some computational overhead due to spectral clustering and sensitivity to IoU thresholds for defining negatives. Future research may focus on automating threshold selection or applying these negative restoration principles to other detection architectures beyond the RepMet pipeline.
Summary updated