Query-Based Hard-Image Retrieval for Object Detection at Test Time

Edward Ayers, Jonathan Sadeghi, John Redford, Romain Mueller, Puneet K. Dokania

TL;DR

This paper introduces a novel framework for identifying "hard" images for object detectors at test time, going beyond generic uncertainty measures. By defining specific "hardness queries" (e.g., "images with many false positives"), their method, "score sampling," effectively retrieves images that challenge a detector according to these custom criteria, without needing ground truth labels during inference. This is crucial for applications like autonomous driving where understanding specific failure modes is paramount.

What Problem Does This Paper Address?

Object detection models, while powerful, can still make mistakes. Identifying why and when they fail is critical for improving their reliability, especially in safety-critical applications like autonomous driving. Traditional methods for finding "hard" images often rely on general uncertainty metrics (like entropy) which treat all errors equally. However, not all detection errors are created equal; a missed pedestrian is far more serious than a missed distant car. The challenge is to move beyond generic uncertainty and pinpoint images that are difficult for specific, user-defined reasons, without requiring expensive ground truth annotations at inference time.

Methods

The core of the paper is the Query-Based Hardness framework. Instead of a single definition of "hard," they propose allowing users to define specific "hardness queries." These queries leverage fine-grained error information (like false positives and false negatives) to characterize what makes an image difficult.

To overcome the lack of ground truth at test time, they introduce Score Sampling. This method treats the detector's confidence scores as parameters for a stochastic model. By sampling from this model, they generate "pseudo ground-truth" instances. These pseudo-ground truths are then used to estimate the expected hardness of an image according to the defined query, using a Monte Carlo approach. This allows for a post-hoc, detector-agnostic estimation of query-based hardness.

Datasets

The authors evaluate their method on two widely used datasets:

  • COCO: (Common Objects in Context) A large-scale dataset for object detection.
  • nuImages: A multimodal dataset for autonomous driving.

They use pre-trained models from these datasets, including RetinaNet, Faster-RCNN, Mask-RCNN, and Cascade Mask-RCNN. For nuImages, they remap class labels to a simplified "pedestrian" and "vehicle" schema to better align with autonomous driving scenarios.

Key Insights and Ideas

  • Hardness is Context-Dependent: The paper strongly emphasizes that what constitutes a "hard" image is not universal but depends on the specific requirements of the application and the detector's failure modes.
  • Beyond Generic Uncertainty: Standard uncertainty measures (entropy, Dempster-Shafer) are "agnostic" to specific error types. The proposed query-based approach allows for fine-grained analysis, revealing that different queries lead to vastly different rankings of "hard" images.
  • Score Sampling as a Proxy: Using detection confidence scores to stochastically generate pseudo ground truths is an effective way to approximate expected hardness without actual ground truth. This is a key enabler for test-time analysis.
  • Practicality for Real-World Systems: The post-hoc and annotation-free nature of their method makes it directly applicable to deployed systems where retraining or extensive labeling is infeasible.

Key Results

  • Superior Ranking Performance: Their "Score Sampling" method consistently outperforms entropy and Dempster-Shafer baselines in ranking images by hardness, achieving significantly higher Normalized Discounted Cumulative Gain (nDCG) scores across various hardness queries. For instance, for the PixelAdj(fp) query, their method achieved an nDCG of 0.86, compared to 0.66 for the baselines.
  • Effective Hard Image Identification: Qualitatively, the hardest images identified by their method align well with human intuition for specific queries, whereas baseline methods produce less relevant results.
  • Robustness to Detector Calibration: Despite detectors often being overconfident (miscalibrated), their method still performs effectively, suggesting that the relative ordering of scores is more important than absolute calibration for their approach.
  • Efficiency: The method requires a relatively small number of Monte Carlo samples (around 10) to achieve stable results, balancing accuracy and computational cost.

Limitations and Future Work

  • Detector Calibration Assumption: The method relies on the assumption that detectors are reasonably well-calibrated. While experiments show it works even with miscalibrated detectors, further exploration into the impact of calibration is suggested.
  • Stochastic Model Complexity: The current stochastic model is simple. Future work could explore more complex models for potentially improved ranking performance.
  • Out-of-Domain Data: The efficacy of the method on out-of-domain data is an area for future investigation.
  • More Expressive Queries: While the framework supports a wide range of queries, exploring even more complex and nuanced definitions of hardness is a potential avenue.

Summary updated

Abstract

There is a longstanding interest in capturing the error behaviour of object detectors by finding images where their performance is likely to be unsatisfactory. In real-world applications such as autonomous driving, it is also crucial to characterise potential failures beyond simple requirements of detection performance. For example, a missed detection of a pedestrian close to an ego vehicle will generally require closer inspection than a missed detection of a car in the distance. The problem of predicting such potential failures at test time has largely been overlooked in the literature and conventional approaches based on detection uncertainty fall short in that they are agnostic to such fine-grained characterisation of errors. In this work, we propose to reformulate the problem of finding "hard" images as a query-based hard image retrieval task, where queries are specific definitions of "hardness", and offer a simple and intuitive method that can solve this task for a large family of queries. Our method is entirely post-hoc, does not require ground-truth annotations, is independent of the choice of a detector, and relies on an efficient Monte Carlo estimation that uses a simple stochastic model in place of the ground-truth. We show experimentally that it can be applied successfully to a wide variety of queries for which it can reliably identify hard images for a given detector without any labelled data. We provide results on ranking and classification tasks using the widely used RetinaNet, Faster-RCNN, Mask-RCNN, and Cascade Mask-RCNN object detectors. The code for this project is available at https://github.com/fiveai/hardest.
BibTeX
@inproceedings{ayers2023query,
  author = {Edward Ayers and Jonathan Sadeghi and John Redford and Romain Mueller and Puneet K. Dokania},
  title = {{Query-Based Hard-Image Retrieval for Object Detection at Test Time}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=XhGhNpznzr7},
}
Figure