Decoupled PROB: Decoupled Query Initialization Tasks and Objectness-Class Learning for Open World Object Detection

Riku Inoue, Masamitsu Tsuchiya, Yuji YasuiWACV 2025

TL;DR

This paper introduces "Decoupled PROB," a novel approach to Open World Object Detection (OWOD) that tackles the challenge of distinguishing known from unknown objects. By decoupling the learning of objectness and class predictions and introducing a smarter query initialization strategy, Decoupled PROB significantly improves detection performance on challenging OWOD benchmarks.

What Problem Does This Paper Address?

Traditional object detectors are trained on a fixed set of classes. However, in real-world applications like autonomous driving, detectors frequently encounter objects they haven't seen during training (unknown objects). Open World Object Detection (OWOD) aims to address this by enabling detectors to identify and classify both known and unknown objects, and crucially, to incrementally learn new classes without forgetting previously learned ones. A key challenge in OWOD is the absence of ground truth labels for unknown objects, making it difficult to train models effectively. Existing methods often rely on generating "pseudo-labels" for unknown objects, which can be error-prone. While some recent methods like PROB [43] avoid pseudo-labeling by predicting "probabilistic objectness," they can suffer from conflicting learning signals between objectness and class predictions. This paper aims to resolve these conflicts and improve the ability of models to detect unknown objects.

Methods

The proposed "Decoupled PROB" builds upon the PROB framework and introduces two key innovations:

  • Task-Decoupled Query Initialization (TDQI): This module addresses how object queries (the initial representations that the model uses to find objects) are initialized. Instead of relying solely on learnable queries that spread across the entire image (which can be inefficient for known objects) or query selection based on class scores (which can miss unknown objects), TDQI combines both. It uses query selection to focus on known objects and learnable queries to cover potential unknown objects. This allows the model to efficiently extract features from object regions in the early stages of the decoder, crucial for accurate objectness prediction.
  • Early Termination of Objectness Prediction (ETOP): This technique aims to mitigate the learning conflicts between objectness and class predictions. In PROB, objectness, class, and bounding box are predicted at each decoder layer. ETOP stops the objectness prediction in the shallower layers of the decoder, while continuing to predict class and bounding box throughout all layers. This prevents objectness and class predictions from interfering with each other during training.

Datasets

The paper evaluates its method on two OWOD benchmarks:

  • M-OWODB (Multi-Superclass OWOD Benchmark): This benchmark combines images from MS-COCO and Pascal VOC 2007, divided into four tasks where new classes are introduced incrementally.
  • S-OWODB (Superclass-Separated OWOD Benchmark): This benchmark also uses MS-COCO but groups object categories into distinct super-categories for a clearer separation of learning.

The paper also mentions experiments on the Pascal VOC 2007 dataset for incremental object detection (iOD) tasks with different settings (10+10, 15+5, and 19+1 classes).

Key Insights and Ideas

  • The importance of query initialization for OWOD: Simply applying query selection, a technique successful in closed-set detection, to OWOD models can actually hurt performance. This is because class-score-based query selection is inherently biased towards known objects and struggles to capture unknown ones.
  • Decoupling is key to resolving learning conflicts: The paper highlights that simultaneously predicting objectness and class in every decoder layer can lead to conflicting learning signals. ETOP effectively addresses this by separating the objectness prediction phase from the class and bounding box refinement phase.
  • TDQI offers a balanced approach: By combining the strengths of query selection (focusing on known objects) and learnable queries (broad coverage for unknown objects), TDQI provides a more robust initialization strategy for OWOD.
  • Early object feature extraction is crucial: TDQI's ability to distribute reference points around foreground objects in the initial decoder layers allows for more accurate objectness estimation early in the process, which is vital when objectness prediction is limited to shallow layers.

Key Results

Decoupled PROB demonstrates state-of-the-art performance on both M-OWODB and S-OWODB benchmarks.

  • M-OWODB: Decoupled PROB significantly outperforms the baseline PROB across various metrics, showing notable improvements in "Current known mAP" and "Both mAP" in later tasks.
  • S-OWODB: Decoupled PROB also achieves superior results, particularly in "Current known mAP" and "Both mAP," outperforming PROB by substantial margins in later tasks.
  • Qualitative Results: Visualizations show Decoupled PROB correctly identifying objects as known classes that PROB misclassifies as unknown. It also successfully detects unknown objects that PROB misses entirely.
  • Incremental Object Detection: On Pascal VOC 2007, Decoupled PROB shows favorable performance compared to existing methods and the PROB baseline.

Limitations and Future Work

  • Limitations of Query Selection in OWOD: The paper explicitly shows that simply applying query selection to OWOD models can lead to decreased performance, highlighting its limitations in handling unknown objects.
  • Future Work: While not explicitly detailed, the authors' introduction of TDQI as a module that can be easily integrated into other Deformable DETR-based OWOD models suggests potential for broader application and further research in combining these modules with other advanced OWOD architectures. The success of Decoupled PROB on challenging benchmarks also opens avenues for exploring its application in more complex real-world scenarios.

Summary updated

BibTeX
@inproceedings{inoue2025decoupled,
  author = {Riku Inoue and Masamitsu Tsuchiya and Yuji Yasui},
  title = {{Decoupled PROB: Decoupled Query Initialization Tasks and Objectness-Class Learning for Open World Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure