Semi-Supervised Object Detection in the Open World

Garvita Allabadi, Ana Lucic, Peter Pao-Huang, Yu-Xiong Wang, Vikram S. Adve

Introduction

Standard semi-supervised object detection (SSOD) models typically operate under a "closed-world" assumption, where labeled and unlabeled datasets are expected to share the same set of categories. However, real-world unlabeled data often contains "out-of-distribution" (OOD) objects—classes previously unseen by the model. When encountered, these objects often lead to "semantic expansion," where the model incorrectly force-fits unknown objects into known categories, generating noisy pseudo-labels that degrade overall performance.

This paper introduces a framework for Generalized Open-World Semi-Supervised Object Detection (OWSSD). Instead of simply filtering out OOD data, the proposed method identifies these objects as "unknown" and incorporates them into the learning process, enhancing the robustness of the model for both known and novel categories.

Methodology: The OOD-Aware Framework

The authors propose a two-stage pipeline centered around an OOD Explorer and a Teacher-Student learning mechanism.

The OOD Explorer

The OOD Explorer is designed to handle two critical tasks:

  1. Classification via Ensemble Autoencoders: Rather than a single classifier, the framework employs an ensemble of autoencoders, each specialized for a specific "in-distribution" (ID) class. Objects with high reconstruction errors across all autoencoders are flagged as OOD.
  2. Unsupervised Localization: To overcome the bias of standard Region Proposal Networks (RPNs) toward known classes, the framework utilizes CutLER, an unsupervised, class-agnostic detector. This allows the model to identify potential OOD objects based on visual boundaries rather than semantic labels.
Localization of a potential OOD object (Dog) using class-agnostic detection.

Teacher-Student Pipeline

The training proceeds in two stages. First, a "Teacher" model is trained on a small labeled ID dataset. In the second stage, the Teacher generates pseudo-labels for ID objects in the unlabeled data, while the OOD Explorer identifies "unknown" objects. These labels are fused, prioritizing OOD labels in cases of high overlap to prevent misclassification, and used to train a "Student" model on the entire dataset.

Experimental Setup

The framework was evaluated using the PASCAL VOC dataset. To simulate an open-world environment, a VOC-15 split was created:

  • ID Classes: 15 categories from the VOC07 trainval set.
  • OOD Classes: 5 categories from the VOC20 set appearing only in the unlabeled VOC12 trainval set.
  • Evaluation: Conducted on the VOC07 test set containing all 20 classes.

Results and Key Insights

The proposed method demonstrated significant improvements over standard SSOD baselines:

  • Improved ID Robustness: The framework achieved a 64.7 AP50 for ID classes on the VOC-15 split, outperforming both supervised-only (61.3) and standard SSOD (64.3) models.
  • OOD Discovery: The model successfully localized and identified OOD objects with an 8.8 AP50 (reaching 10.2 in specific configurations), a capability entirely absent in traditional SSOD models.
  • Effective OOD Detection: The OOD Explorer outperformed common methods like Isolation Forests or KNN, achieving an AUROC of 0.72.
Detection of an airplane, demonstrating class-agnostic localization.

A key finding is that explicitly labeling OOD objects as "unknown" prevents "semantic drift," where the model's understanding of known classes is corrupted by incorrectly labeled OOD samples. Furthermore, the use of unsupervised localization like CutLER is vital, as standard detectors are often "blind" to objects outside their training distribution.

Discussion and Future Work

While the framework marks a significant step forward, localization remains a primary challenge. The gap between current performance and an "Oracle" proposer (which would jump OOD AP50 from ~10 to ~36) suggests that more precise class-agnostic localization techniques are needed. Future research will likely explore continual learning, enabling models to eventually differentiate between various "unknown" classes without human intervention.

Summary updated

BibTeX
@inproceedings{allabadi2024semi,
  author = {Garvita Allabadi and Ana Lucic and Peter Pao-Huang and Yu-Xiong Wang and Vikram S. Adve},
  title = {{Semi-Supervised Object Detection in the Open World}},
  booktitle = {OWA @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=1rs9HXmJUv},
}
Figure