KTCN: Enhancing Open-World Object Detection with Knowledge Transfer and Class-Awareness Neutralization

Xing Xi, Yangyang Huang, Jinhao Lin, Ronghua Luo

Abstract

Open-World Object Detection (OWOD) has garnered widespread attention due to its ability to recall unannotated objects. Existing works generate pseudo-labels for the model using heuristic priors, which limits the model’s performance. In this paper, we leverage the knowledge of the large-scale visual model to provide supervision for unknown categories. Specifically, we use the Segment Anything Model (SAM) to generate raw pseudo-labels for potential objects and refine them through Intersection over Union (IOU) and the shortest bounding box side length. Nevertheless, the abundance of pseudo-labels still exacerbates the competition issue in the one-to-many label assignment. To address this, we propose the Dual Matching Label Assignment (DMLA) strategy. Furthermore, we propose the Class-Awareness Neutralizer (CAN) to reduce the model’s bias towards known categories. Evaluation results on open-world object detection benchmarks, including MS COCO and Pascal VOC, show that our method achieves nearly 200% the unknown recall rate of previous state-of-the-art (SOTA) methods, reaching 41.5 U-Recall. Additionally, our approach does not add any extra parameters, maintaining the inference speed advantage of Faster R-CNN, leading the SOTA methods based on deformable DETR at a speed of over 10 FPS.

Introduction

Traditional object detection operates under a "closed-world" assumption, where models are trained and evaluated on a fixed set of categories. In contrast, Open-World Object Detection (OWOD) requires models to identify known classes while simultaneously detecting "unknown" objects. These unknown instances can then be labeled and integrated into the model incrementally.

Despite its potential, current OWOD research faces three primary hurdles: low-quality pseudo-labels derived from simple heuristics, label competition between ground-truth and pseudo-labels during training, and an inherent class bias where models favor known categories over unknown ones. This paper introduces the KTCN framework to address these challenges by leveraging foundation models and novel architectural strategies.

Methodology

The KTCN architecture integrates SAM-based knowledge transfer, Dual Matching Label Assignment (DMLA), and the Class-Awareness Neutralizer (CAN).

Knowledge Transfer from SAM

To overcome the noise of heuristic-based mining, KTCN utilizes the Segment Anything Model (SAM) to generate high-quality pseudo-labels. SAM provides class-agnostic masks for potential objects, which are converted into bounding boxes. These candidates are then filtered using IoU and side-length constraints to ensure they represent distinct, plausible unknown objects.

Dual Matching Label Assignment (DMLA)

Introducing pseudo-labels often leads to competition for anchor boxes. DMLA resolves this through a hierarchical matching process. Predictions are first matched to known ground-truth labels. Only the remaining, unmatched predictions are eligible for matching with SAM-generated pseudo-labels. This ensures that the supervision of known classes remains stable while expanding the model's ability to detect unknowns.

Class-Awareness Neutralizer (CAN)

Models are naturally biased toward known categories, often suppressing unknown detections. CAN is an inference-time correction mechanism that utilizes the class-agnostic objectness (or centerness) score from the Region Proposal Network (RPN). By adjusting the final classification confidence with this geometric signal, CAN boosts the scores of potential unknown objects that the classifier might otherwise overlook due to lack of explicit class-specific training.

Experimental Results

Performance Benchmarks

KTCN was evaluated using a hybrid MS COCO and Pascal VOC dataset across four incremental tasks ($T_1$ to $T_4$). The results demonstrate a significant breakthrough in unknown object detection.

Comparison of U-Recall and mAP across incremental tasks shows KTCN (Ours) significantly outperforming existing SOTA methods.

As shown in the performance charts, KTCN achieved an Unknown Recall (U-Recall) of 41.5%, nearly doubling the performance of previous state-of-the-art methods like PROB (19.4%) and CAT (23.7%). Crucially, this gain does not come at the expense of known class accuracy; KTCN maintained a high mAP of 60.2 on Task 1, outperforming all baselines.

Efficiency

By building upon the Faster R-CNN architecture without adding extra parameters for inference, KTCN maintains high efficiency. It achieves a processing speed of 36.36 FPS, significantly faster than recent DETR-based models which typically operate below 20 FPS. This makes KTCN a practical solution for real-time open-world applications.

Conclusion

KTCN demonstrates that transferring knowledge from large-scale foundation models like SAM is a highly effective strategy for Open-World Object Detection. By combining high-quality pseudo-labels with a prioritized matching strategy (DMLA) and an inference-time bias neutralizer (CAN), the framework achieves state-of-the-art recall for unknown objects while remaining computationally efficient. Future work will explore further distillation techniques to maintain high unknown recall as the pool of known classes grows during incremental learning tasks.

Summary updated

Abstract

Open-World Object Detection (OWOD) has garnered widespread attention due to its ability to recall unannotated objects. Existing works generate pseudo-labels for the model using heuristic priors, which limits the model’s performance. In this paper, we leverage the knowledge of the large-scale visual model to provide supervision for unknown categories. Specifically, we use the Segment Anything Model (SAM) to generate raw pseudo-labels for potential objects and refine them through Intersection over Union (IOU) and the shortest bounding box side length. Nevertheless, the abundance of pseudo-labels still exacerbates the competition issue in the one-to-many label assignment. To address this, we propose the Dual Matching Label Assignment (DMLA) strategy. Furthermore, we propose the Class-Awareness Neutralizer (CAN) to reduce the model’s bias towards known categories. Evaluation results on open-world object detection benchmarks, including MS COCO and Pascal VOC, show that our method achieves nearly 200% the unknown recall rate of previous state-of-the-art (SOTA) methods, reaching 41.5 U-Recall. Additionally, our approach does not add any extra parameters, maintaining the inference speed advantage of Faster R-CNN, leading the SOTA methods based on deformable DETR at a speed of over 10 FPS. Our code is available at https://github.com/xxyzll/KTCN.
BibTeX
@inproceedings{xi2024ktcn,
  author = {Xing Xi and Yangyang Huang and Jinhao Lin and Ronghua Luo},
  title = {{KTCN: Enhancing Open-World Object Detection with Knowledge Transfer and Class-Awareness Neutralization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=FJ86t62tHM},
}
Figure