Introspective False Negative Prediction for Black-Box Object Detectors in Autonomous Driving

Qinghua Yang, Hui Chen, Zhe Chen, Junzhe Su2021 · DOI

TL;DR

Autonomous driving systems rely heavily on object detectors, but these can miss crucial objects (False Negatives or FNs). This paper introduces an "introspective" framework that learns to predict these missed objects without needing access to the detector's internal workings. This approach significantly improves detection recall and overall safety by flagging potential failures before they happen.

What Problem Does This Paper Address?

Object detection is a cornerstone of autonomous driving, enabling vehicles to perceive their surroundings. However, even state-of-the-art detectors are not perfect and can fail in various ways. Two common failure modes are False Positives (detecting non-existent objects) and False Negatives (missing actual objects). While FPs can lead to unnecessary braking or evasive maneuvers, FNs are far more dangerous, potentially causing catastrophic accidents by failing to detect pedestrians, other vehicles, or obstacles.

Current approaches to mitigate these failures often focus on improving the detector's inherent accuracy or predicting general uncertainty about the scene. However, these methods often fail to address FNs specifically, or they rely on handcrafted features that are not universally applicable or may not capture the true reasons for a detection failure. Furthermore, many existing methods require access to the internal architecture of the object detector, limiting their applicability to "black-box" detectors where internal details are unknown. This paper tackles the critical problem of reliably predicting False Negatives in real-time for any object detector, even when its internal workings are inaccessible.

Methods

The core innovation is an "introspective" framework that treats FN prediction as a self-assessment task for the object detection system. Instead of relying on predefined rules or handcrafted features, the framework trains a Convolutional Neural Network (CNN) to learn what constitutes an FN for a specific black-box detector.

Key methodological components include:

  • Introspective FN Predictor: A CNN-based model designed to extract "FN features" directly from input images. This predictor is trained to identify patterns that typically lead to missed detections by the target object detector.
  • Object-wise Prediction: Unlike previous methods that might predict a per-image failure probability, this framework is designed to output object-wise predictions. This means it can identify the location and category of potential FNs, providing actionable information.
  • Multi-branch Cooperation Mechanism: To combat the severe foreground-background imbalance inherent in identifying missed objects (where the "missed" object is a small part of a large image), the predictor uses multiple branches. One branch predicts a segmentation mask to highlight potential FN regions, effectively narrowing down the search space for other branches. This mechanism helps the model focus on relevant areas and learn more robust features.
  • Black-Box Compatibility: The framework is designed to work with any object detector without needing access to its internal architecture or parameters. This is achieved by training the introspective predictor on the outputs of the black-box detector and the ground truth.

Datasets

The paper utilizes several datasets for training and evaluation:

  • KITTI 2D Object Dataset: Used to construct the KITTI 2D FN dataset. This dataset was created by running a pre-trained SSD detector on KITTI images and identifying the missed car objects as FNs. This dataset was used for training and validation.
  • KITTI Tracking Dataset (TRK): Used to construct the TRK-FN dataset. Similar to the 2D-FN dataset, this was created by evaluating an SSD detector on the TRK dataset and extracting FN car labels. This dataset was used for testing.
  • nuScenes Dataset: A large-scale, real-world urban driving dataset used for testing to evaluate the framework's performance on data with significant domain shifts compared to the training data.
  • GTA92 Dataset: A synthetic dataset generated from a game engine, also used for testing to assess performance on out-of-distribution data.

Key Insights and Ideas

  • FNs are Detector-Specific: The paper highlights that FNs are not universal. Different object detectors miss different objects in the same scene, underscoring the need for a detector-specific FN prediction approach.
  • Handcrafted Features are Insufficient: Empirical observations show that common FN-related factors (like occlusion or truncation) only explain a portion of FN instances. This justifies the move towards learning FN features intrinsically.
  • Introspection as a Paradigm Shift: Framing FN prediction as an "introspection" task—where a system learns to recognize its own potential failures—is a novel and powerful approach, especially for black-box systems.
  • Multi-branch Cooperation for Imbalance: The multi-branch approach, particularly the segmentation branch, is crucial for handling the extreme foreground-background imbalance in FN detection, allowing the model to focus effectively.

Key Results

The proposed framework demonstrates strong performance:

  • Bounding Box Level: Achieves 81.95% precision and 88.10% recall on the 2D-FN test dataset with a 0.5 IOU threshold, correctly identifying a significant portion of missed objects.
  • Pixel Level: Reports 71.71% Precision and 83.47% Recall on the pixel level, indicating accurate localization of FN pixels.
  • Quantitative Improvements to Object Detection: When FN predictions are incorporated, the framework significantly improves the recall, AP, and F1 score of the base object detector. For example, on the 2D-FN test dataset, recall increases by 54.24% and AP by 51.44%.
  • Domain Adaptation: The framework shows promising results even on datasets (nuScenes, GTA92) with significant domain disparities from the training data, indicating good generalization capabilities.

Limitations and Future Work

  • Precision Improvement: While recall and AP see substantial gains, the precision of the base object detector is only marginally improved. The authors attribute this to the addition of predicted FN bounding boxes, which can sometimes be false positives themselves. They suggest that more advanced post-processing methods might be needed.
  • Input Resolution: The paper notes that higher input resolutions generally lead to better FN prediction performance, but this comes at the cost of increased computational resources and potentially slower training.
  • Future Work: The authors plan to incorporate uncertainty estimation into their FN predictions and integrate this with the object-wise decoder to further enhance accuracy. They also aim to explore more sophisticated post-processing techniques to address the precision limitations.

Summary updated

BibTeX
@misc{yang2021introspective,
  author = {Qinghua Yang and Hui Chen and Zhe Chen and Junzhe Su},
  title = {{Introspective False Negative Prediction for Black-Box Object Detectors in Autonomous Driving}},
  year = {2021},
  doi = {10.3390/s21082819},
}
Figure