Class-Agnostic Object Detection

Ayush Jaiswal, Yue Wu, Pradeep Natarajan, Premkumar NatarajanWACV 2021

TL;DR

This paper introduces "class-agnostic object detection," a new paradigm that aims to detect all objects in an image, regardless of their known class, addressing a key limitation of traditional detectors. They propose an adversarial training framework that encourages models to learn general objectness features, leading to improved generalization to unseen object types and better downstream utility.

What Problem Does This Paper Address?

Traditional object detection models are trained to identify and classify objects belonging to a predefined set of categories. This approach faces several challenges:

  • Data Annotation Cost: Creating large-scale datasets with annotations for every possible object type is prohibitively expensive and time-consuming. Consequently, real-world datasets often only contain annotations for a limited number of "objects of interest," treating everything else as background.
  • Limited Generalization: Models trained on these limited datasets struggle to detect objects of unknown or novel types. This severely restricts their applicability in scenarios where encountering unexpected objects is common, such as autonomous driving (obstacle detection) or large-scale image retrieval.
  • Background Bias: The standard evaluation metrics, like mean average precision (mAP), implicitly penalize the detection of unknown objects, further reinforcing the focus on known categories.

The authors propose "class-agnostic object detection" as a solution, aiming to predict bounding boxes for all objects in an image, irrespective of their class. This allows for downstream systems to perform application-specific classification or retrieval on detected objects, even those not seen during training.

Methods

The core of the paper's contribution lies in its novel adversarial training framework designed to achieve class-agnostic detection. The approach involves:

  • Problem Formulation: Defining class-agnostic object detection as predicting bounding boxes for all objects, without predicting their class labels.
  • Baseline Models: The authors establish baseline performance using standard object detection components:
  • Region Proposal Networks (RPNs) from two-stage detectors.
  • Class-aware detectors trained for object-or-not classification.
  • Pretrained class-aware models fine-tuned for binary classification.
  • Models trained from scratch for binary classification.
  • Adversarial Framework: This is the key innovation. The framework augments a standard object detector with an adversarial object-type discriminator.
  • Upstream Module: This part of the detector (e.g., backbone, RPN) extracts features from the image.
  • Discriminator: This module attempts to predict the object type from the features extracted by the upstream module.
  • Adversarial Training: The detector is trained to fool the discriminator. This is achieved by minimizing a combined loss:
  • Object-or-not Classification Loss: Standard binary classification loss.
  • Bounding Box Regression Loss: Standard regression loss.
  • Negative Entropy of Discriminator Predictions: This term penalizes the upstream module if its features contain object-type information that the discriminator can exploit. By maximizing the entropy of the discriminator's predictions, the model is forced to produce features that are ambiguous with respect to object class.
  • Training Procedure: The training involves an alternating process where the discriminator is updated multiple times for each update of the main detector modules.

The framework is instantiated for both two-stage (Faster R-CNN) and one-stage (SSD) detectors.

Datasets

The paper utilizes several standard datasets for training and evaluation:

  • PASCAL VOC: Used for training and evaluating generalization to unseen classes within the same dataset.
  • MS COCO: Used for training and evaluating generalization to unseen classes.
  • Open Images V6: Used for evaluating generalization to unseen classes.
  • ObjectNet: Used to evaluate the downstream utility of the detected bounding boxes for object recognition.

The authors also propose specific splits and protocols for evaluating generalization to unseen object types.

Key Insights and Ideas

  • The "Objectness" Problem: The paper reframes object detection from "what is this object?" to "is this a bounding box around any object?". This shift is crucial for handling novelty.
  • Adversarial Training for Feature Disentanglement: The adversarial discriminator acts as a regularizer, forcing the feature extractor to discard class-specific information and retain only general objectness cues. This is a powerful technique for learning more robust and generalizable representations.
  • Generalization is Key: The results strongly suggest that models that generalize better to unseen object types (i.e., are more "class-agnostic") also exhibit higher downstream utility, indicating that robust object localization is a prerequisite for many advanced vision tasks.
  • Scratch vs. Fine-tuning: Models trained from scratch for class-agnostic detection generally outperform those fine-tuned from pretrained class-aware models, suggesting that fine-tuning might retain some class-specific biases that hinder generalization.

Key Results

  • Improved Generalization: Class-agnostic models, particularly those trained adversarially, consistently outperform class-aware baselines in detecting unseen object types across PASCAL VOC, COCO, and Open Images datasets.
  • Adversarial Advantage: The adversarial training framework shows significant gains, especially on "hard" unseen classes, demonstrating its effectiveness in learning general objectness.
  • Downstream Utility: Adversarially trained class-agnostic detectors achieve better object recognition accuracy on the ObjectNet dataset when using the predicted bounding boxes, highlighting the practical benefits of the proposed approach. For instance, FRCNN-ag-ad models showed improved BO-acc and Acc@M compared to FRCNN-aw.
  • Model Agnostic: The adversarial framework is shown to be effective for both two-stage (Faster R-CNN) and one-stage (SSD) detectors.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that fine-tuning pretrained class-aware models for class-agnostic detection can be less effective than training from scratch, suggesting a difficulty in "unlearning" class-specific features.
  • While the adversarial approach improves generalization, there's a slight drop in performance on "seen" classes for some models, indicating a potential trade-off.
  • Future Work:
    • Further refinement of the class-agnostic object detection problem formulation.
  • Development of more sophisticated training and evaluation protocols.
  • Exploration of novel methods for class-agnostic object detection beyond the proposed adversarial framework.
  • Investigating the trade-off between generalization to unseen classes and performance on seen classes.

Summary updated

BibTeX
@inproceedings{jaiswal2021class,
  author = {Ayush Jaiswal and Yue Wu and Pradeep Natarajan and Premkumar Natarajan},
  title = {{Class-Agnostic Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure