Abstract
In many applications, such as autonomous driving, hand manipulation, or robot navigation, object detection methods must be able to detect objects unseen in the training set. Open World Detection (OWD) seeks to tackle this problem by generalizing detection performance to seen and unseen class categories. Recent works have seen success in the generation of class-agnostic proposals, which we call Open-World Proposals (OWP), but this comes at the cost of a big drop on the classification task when both tasks are considered in the detection model. These works have investigated two-stage Region Proposal Networks (RPN) by taking advantage of objectness scoring cues; however, for its simplicity, run-time, and decoupling of localization and classification, we investigate OWP through the lens of fully convolutional one-stage detection network, such as FCOS. We show that our architectural and sampling optimizations on FCOS can increase OWP performance by as much as 6% in recall on novel classes, marking the first proposal-free one-stage detection network to achieve comparable performance to RPN-based two-stage networks. Furthermore, we show that the inherent, decoupled architecture of FCOS has benefits to retaining classification performance. While two-stage methods worsen by 6% in recall on novel classes, we show that FCOS only drops 2% when jointly optimizing for OWP and classification.
Introduction
Standard object detection operates under a "closed-world" assumption, where models are trained and evaluated on a fixed set of categories. However, real-world deployment in robotics or autonomous systems requires the ability to handle "unknown" objects. Open World Detection (OWD) addresses this by enabling models to detect objects they haven't seen during training. A foundational component of OWD is Open-World Proposals (OWP)—the generation of class-agnostic bounding boxes for any potential object in a scene.
Traditionally, two-stage detectors like Faster R-CNN have been preferred for OWP due to their explicit objectness scoring in the Region Proposal Network (RPN). However, these models often suffer significant performance degradation in their primary classification task when optimized for open-world discovery. This work investigates the potential of one-stage detectors, specifically FCOS (Fully Convolutional One-Stage Object Detection), to provide high-quality OWP while maintaining robust classification performance.
Methodology
The proposed approach enhances the FCOS architecture through three primary optimizations designed to improve "objectness" estimation and generalization.
IOU Overlap as Objectness
While standard FCOS utilizes a "centerness" score to rank proposals, this metric primarily filters low-quality boxes near object edges rather than assessing absolute localization accuracy. We propose using the predicted Intersection over Union (IOU) between predicted and ground-truth boxes as a superior signal for objectness.
To further refine this, we introduce a Conditional IOU Head. This architectural modification feeds the output of the regression branch back into the IOU and Centerness branches, providing essential geometric context for more accurate scoring.
IOU-Based Sampling
Effective training of the objectness branch requires a diverse set of examples. We implement a novel sampling strategy that moves beyond center-pixel selection. By explicitly sampling "hard negatives"—pixels with low IOU relative to ground truth—the model learns to better discriminate between valid objects and complex background regions.
Unknown Object Masking
A common challenge in OWD is that unlabeled objects in the training set are often treated as background, confusing the model. We utilize the objectness branch to generate a mask that filters out high-confidence "unknown" regions during classification training, preventing the model from incorrectly learning to ignore potential novel objects.
Experimental Evaluation
The model was evaluated on the COCO and LVIS datasets using a "base" vs. "novel" class split to test generalization.
Key Results
- Recall Improvements: The optimized FCOS achieved a 6% increase in recall on novel classes compared to the baseline.
- Competitive Performance: This marks the first one-stage, proposal-free network to match the performance of RPN-based two-stage networks in OWP tasks.
- Classification Retention: Unlike two-stage methods, which saw a 6.2% drop in novel class recall when jointly optimized, our FCOS-based method only dropped 2%. This demonstrates that the decoupled nature of one-stage heads is inherently better suited for balancing discovery and classification.
Conclusion
By leveraging the decoupled architecture of one-stage detectors and introducing IOU-based scoring and sampling, we demonstrate that FCOS can serve as a powerful engine for Open-World Proposals. Our method achieves state-of-the-art recall on unseen objects while significantly outperforming two-stage alternatives in retaining classification accuracy. Future work will explore more sophisticated masking techniques to further improve performance on sparse datasets like LVIS.
Summary updated