KTCN: Enhancing Open-World Object Detection with Knowledge Transfer and Class-Awareness Neutralization

Xing Xi, Yangyang Huang, Jinhao Lin, Ronghua Luo, Yupei Wu, Tao Deng, Mingjie Sun

Abstract

Open-World Object Detection (OWOD) presents a significant challenge for computer vision: models must not only recognize a set of known categories but also identify and flag "unknown" objects for future incremental learning. Current OWOD methods often suffer from weak supervision for unknown classes and a strong classification bias toward known categories. We introduce KTCN (Knowledge Transfer and Class-Awareness Neutralization), a framework designed to bridge these gaps. By leveraging the Segment Anything Model (SAM) as a teacher, KTCN generates high-quality pseudo-labels for potential unknown objects. We propose a Dual Matching Label Assignment (DMLA) strategy to prevent pseudo-labels from interfering with ground-truth learning, and a Class-Awareness Neutralizer (CAN) to mitigate inference-time bias using class-agnostic objectness scores. Our approach achieves a nearly 200% increase in unknown object recall compared to previous state-of-the-art methods while maintaining the real-time performance of a Faster R-CNN backbone.

The Open-World Challenge

Standard object detection operates under a "closed-world" assumption, where models are trained and tested on a fixed set of categories. In dynamic real-world environments, such as autonomous driving or robotics, this assumption fails. Open-World Object Detection (OWOD) requires a model to perform three tasks: identify known objects, flag unknown objects as a generic "unknown" class, and incrementally learn these new objects when labels are provided.

Previous research in OWOD has been hindered by three primary issues:

  1. Weak Supervision: Relying on simple heuristics to identify unknown objects leads to noisy training data.
  2. Label Competition: Pseudo-labels for unknowns often overlap with known ground-truth, causing the model to struggle with conflicting signals.
  3. Class Bias: Models naturally favor known classes, often suppressing the detection of unknown objects during inference.

Methodology: The KTCN Framework

KTCN addresses these limitations through a multi-stage pipeline that integrates foundation model knowledge with architectural innovations.

Knowledge Transfer via SAM

Instead of relying on heuristics, KTCN uses the Segment Anything Model (SAM) to generate "raw" pseudo-labels. SAM's massive pre-training allows it to identify almost any object in an image. These raw boxes are filtered based on their Intersection over Union (IoU) with known instances and their physical size to ensure high-quality supervision for the "unknown" class.

Dual Matching Label Assignment (DMLA)

To resolve label competition, KTCN introduces a two-stage matching process. In the first stage, predictions are matched against high-certainty ground-truth labels for known classes. Only the remaining unmatched predictions are then allowed to match with the SAM-generated pseudo-labels. This hierarchy ensures that the model's ability to recognize known objects is never compromised by the inclusion of unknown pseudo-labels.

The KTCN architecture integrates a Dual Matching Label Assignment strategy and a Class-Awareness Neutralizer to improve unknown object detection.

Class-Awareness Neutralizer (CAN)

During inference, standard classifiers often assign very low scores to unknown objects because they don't fit any known category profile. CAN corrects this by leveraging the centerness score from the Region Proposal Network (RPN). Since the RPN is class-agnostic—focusing only on whether a "thing" exists—its score provides a reliable measure of objectness. CAN uses this signal to boost the confidence of potential unknown detections, effectively neutralizing the classifier's bias.

Experimental Results

We evaluated KTCN using the standard OWOD protocol across MS COCO and Pascal VOC datasets, split into four incremental tasks ($T_1$ to $T_4$).

Massive Recall Gains

KTCN demonstrates a dramatic improvement in identifying unknown objects. As shown in the performance metrics, KTCN achieved an Unknown Recall (U-Recall) of 41.5, nearly doubling the performance of previous state-of-the-art methods like CAT (23.7) and PROB (19.4). This suggests that the combination of SAM-based supervision and the CAN module is highly effective at capturing objects outside the training set.

Performance charts showing KTCN (red) significantly outperforming existing methods in Unknown Recall across all tasks.

Efficiency and Deployment

Despite its performance gains, KTCN remains highly efficient. By building on the Faster R-CNN architecture and adding zero parameters to the inference pipeline (as CAN is a mathematical adjustment of existing scores), the model maintains a speed of 36.36 FPS. This makes it suitable for real-time applications where heavy Transformer-based models might struggle.

Conclusion

KTCN represents a significant step forward for Open-World Object Detection. By treating large foundation models as teachers and implementing clever architectural safeguards like DMLA and CAN, we can create detectors that are both fast and aware of the "unknown." While there remains a slight trade-off in known-class precision as the world becomes increasingly complex, KTCN's ability to find what it hasn't been taught is currently unparalleled in the field.

Summary updated

BibTeX
@inproceedings{xi2024ktcn,
  author = {Xing Xi and Yangyang Huang and Jinhao Lin and Ronghua Luo and Yupei Wu and Tao Deng and Mingjie Sun},
  title = {{KTCN: Enhancing Open-World Object Detection with Knowledge Transfer and Class-Awareness Neutralization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=FJ86t62tHM},
}
Figure