Tunable Hybrid Proposal Networks for the Open World

Matthew Inkawhich, Nathan Inkawhich, Hai Li, Yiran Chen

Researchers from Duke University and the Air Force Research Laboratory have introduced the Tunable Hybrid Proposal Network (THPN), a novel framework designed to overcome the limitations of the "closed-world" assumption in object detection. While traditional models are trained to recognize a fixed set of classes and ignore everything else as background, THPN provides a flexible mechanism to detect both known objects and novel, unseen entities in open-world environments.

The Challenge of the Closed World

Most modern object detection systems are explicitly conditioned to treat unfamiliar objects as background. This bias prevents them from even proposing regions of interest for novel objects, such as a new vehicle type or specialized equipment. While class-agnostic methods exist, they often sacrifice the precision required for known (In-Distribution, or ID) objects. Furthermore, different deployment scenarios require different trade-offs: a security system might prioritize high recall for any moving object (Out-of-Distribution, or OOD), whereas an industrial sorter requires high precision for specific parts. Existing models lack the flexibility to adapt to these varying requirements.

Methods: Architecture and Self-Training

THPN addresses these challenges through three primary innovations:

1. Adjustable Hybrid Architecture

THPN utilizes two parallel "objectness" heads. One head is classification-based, learning the discriminative features of known classes. The other is localization-based, predicting the quality of a bounding box fit for any potential object. A key feature of this architecture is the hyperparameter $\lambda_{CLS}$, which allows users to interpolate between these two behaviors at inference time.

The THPN architecture featuring parallel classification and localization quality heads controlled by the lambda_CLS parameter.

2. Open-World Aware Self-Training

To move beyond labeled classes, THPN employs an iterative self-training loop. The model generates "pseudo-labels" for unlabeled objects within its training images. These high-confidence guesses are merged back into the training set, teaching the model to recognize general "thing-ness" in a semi-supervised manner.

3. Dynamic Loss Components

To manage the noise inherent in pseudo-labels, the authors developed two specialized loss functions:

  • Localization Quality Focal Loss ($L_{LQF}$): Dynamically weights loss based on the model's prediction of region quality.
  • Weighted Box Regression Loss ($L_{WBR}$): Reduces the influence of bounding box updates from low-confidence pseudo-labels, preventing the model from learning from poor guesses.

Evaluation and Key Results

The authors evaluated THPN across several benchmarks designed to simulate varying degrees of label bias, including VOC, COCO, and specialized tasks like "Animal" (training only on animal classes) and the "Ships Challenge" (detecting civilian ships when trained on military ones).

Open World AR-AUC performance across tasks of increasing difficulty, showing how the optimal lambda_CLS value shifts as the task becomes harder.

Performance Gains

  • Superior Recall: On the VOC $\rightarrow$ COCO benchmark, THPN improved Average Recall (AR100) by +18.9% over the standard Region Proposal Network (RPN) and +5.7% over the previous state-of-the-art, OLN.
  • Robustness to Bias: In the "Animal" task, THPN outperformed baselines by over 11% in OOD recall, demonstrating its ability to generalize far beyond its training categories.
  • Domain Generalization: In satellite imagery ship detection, THPN achieved a ~3x recall improvement over Faster R-CNN for OOD ship types.

Insights and Future Directions

A major finding of the study is the power of tunability. By adjusting $\lambda_{CLS}$, a single model can be optimized for tasks ranging from conservative ID-focused detection to aggressive OOD discovery. Additionally, THPN proved highly data-efficient; a model trained on just 10% of labels could match or exceed standard baselines trained on 100% of the data.

While THPN significantly advances the state of proposal networks, the authors note that open-world detection remains limited by the proposal stage. Future work will focus on integrating THPN into end-to-end detectors and reducing the computational overhead of the iterative self-training process.

Summary updated

Abstract

Current state-of-the-art object proposal networks are trained with a closed-world assumption, meaning they learn to only detect objects of the training classes. These models fail to provide high recall in open-world environments where important novel objects may be encountered. While a handful of recent works attempt to tackle this problem, they fail to consider that the optimal behavior of a proposal network can vary significantly depending on the data and application. Our goal is to provide a flexible proposal solution that can be easily tuned to suit a variety of open-world settings. To this end, we design a Tunable Hybrid Proposal Network (THPN) that leverages an adjustable hybrid architecture, a novel self-training procedure, and dynamic loss components to optimize the tradeoff between known and unknown object detection performance. To thoroughly evaluate our method, we devise several new challenges which invoke varying degrees of label bias by altering known class diversity and label count. We find that in every task, THPN easily outperforms existing baselines (e.g., RPN, OLN). Our method is also highly data efficient, surpassing baseline recall with a fraction of the labeled data.
BibTeX
@inproceedings{inkawhich2024tunable,
  author = {Matthew Inkawhich and Nathan Inkawhich and Hai Li and Yiran Chen},
  title = {{Tunable Hybrid Proposal Networks for the Open World}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00199},
}
Figure