Noise-Aware Evaluation of Object Detectors

Jeffri Murrugarra Llerena, Claudio R. JungWACV 2025

TL;DR

This paper tackles the pervasive issue of noisy annotations in object detection datasets, which can significantly skew evaluation metrics like Average Precision (AP). The authors propose a novel framework to quantify this noise, derive upper and lower bounds for AP, and demonstrate how these bounds reveal the true performance of detectors, especially for small objects and under strict IoU thresholds.

What Problem Does This Paper Address?

Object detection is a cornerstone of many AI applications, from autonomous driving to medical imaging. Its success hinges on large, meticulously annotated datasets. However, human annotation is inherently prone to errors: incorrect labels, missing objects, and, crucially for this paper, imprecise bounding boxes. These "noisy" annotations can lead to misleading evaluations of object detector performance. Standard metrics like Average Precision (AP) rely on Intersection over Union (IoU) to determine if a detection is correct. When the ground truth bounding box itself is slightly off, the IoU can be significantly impacted, leading to an over- or underestimation of the detector's true capabilities. This is particularly problematic for small objects or when using high IoU thresholds, where even minor annotation inaccuracies can lead to drastic changes in AP. The paper aims to provide a more robust evaluation by accounting for this annotation noise.

Methods

The core of the paper's contribution lies in developing a noise-aware evaluation framework. They start by establishing theoretical bounds for any similarity metric that satisfies the triangle inequality (like IoU). For a given predicted bounding box (Det), an annotated bounding box (Ann), and the true ground truth (GT), they show that the true IoU(Det, GT) is bounded by the observed IoU(Det, Ann) plus or minus the "noise distance" between the annotation and the ground truth, denoted as $d_{iou}(Ann, GT)$.

To estimate this noise distance, they leverage a dataset with both original and corrected annotations (from [17]). They focus on the discrepancies in the corner coordinates of the bounding boxes. By analyzing these discrepancies, they model the annotation noise using a truncated Laplace distribution. This distribution is then used to estimate the expected IoU degradation, $\bar{e}(W, H)$, which depends on the dimensions of the bounding box.

Using this estimated noise, they derive noise-aware upper and lower bounds for the AP metric, denoted as $AP^u$ and $AP^l$. These bounds provide a more realistic performance range for object detectors, accounting for the inherent uncertainty in the ground truth annotations.

Datasets

The primary dataset used for analysis and estimation of annotation noise is the Microsoft COCO dataset, specifically utilizing a subset of reannotated bounding boxes provided by [17]. This dataset is crucial as it offers both original and corrected annotations for the same images and objects, allowing for the empirical estimation of noise. The authors also mention using the Open Images Dataset (OID) and DOTA in their discussions and figures to illustrate different types of annotation errors.

Key Insights and Ideas

  • Annotation noise is not uniform: Contrary to some prior assumptions of uniform noise, the paper demonstrates that annotation noise has a more complex distribution, with a sharp peak at the origin and an exponential decay in the tails, well-modeled by a Laplace distribution.
  • Noise impact is size-dependent: The effect of annotation noise on IoU and AP is significantly amplified for smaller objects. This means that standard AP metrics can be highly misleading for small object detection.
  • IoU thresholds matter: The discrepancy between standard AP and the noise-aware bounds is most pronounced at higher IoU thresholds (e.g., 0.95). This highlights the sensitivity of strict evaluation criteria to annotation inaccuracies.
  • AP can be misleadingly optimistic or pessimistic: The noise-aware bounds reveal that detectors can be either over- or underestimated by the standard AP metric. The paper observes a trend where the lower bound ($AP^l$) is often more pessimistic than the upper bound ($AP^u$) is optimistic, suggesting that detectors might be penalized more by noise than they are rewarded.
  • A unified framework: The proposed noise-aware evaluation framework is generalizable and can be applied to different object representations (HBBs, OBBs, 3D) and similarity metrics beyond IoU, provided they satisfy the triangle inequality.

Key Results

The paper evaluates several state-of-the-art object detectors (e.g., Reppoints, YOLOv10, RTMDet, CO-DETR) on the COCO 2017 validation split.

  • For small objects and a restrictive IoU threshold of 0.95, the noise-aware evaluation often results in an AP of 0 for all tested detectors, indicating that achieving such a high IoU with noisy ground truth is practically impossible.
  • The difference between the standard AP and the noise-aware bounds ($AP^u - AP^l$) provides an uncertainty estimate. For instance, for small objects at a 0.5 IoU threshold, this uncertainty can be around 13.5% on average.
  • The paper shows that for YOLOv10 X, the standard AP50 for small objects is 55.6, but the noise-aware bounds are 60.0 ($AP^u$) and 46.3 ($AP^l$), illustrating a significant range of uncertainty.
  • Visualizations in Figure 7 and 8 demonstrate cases where detectors produce visually good detections that might have a lower IoU than expected due to annotation noise, further supporting the need for noise-aware evaluation.

Limitations and Future Work

  • Dataset dependency: The noise distribution was estimated from a specific reannotated subset of COCO. The authors acknowledge that further studies are needed to assess if this noise distribution is generalizable to other datasets.
  • Class-agnostic assumption: The noise model is currently class-agnostic, meaning it assumes the same noise distribution across all object classes. While findings suggest some uniformity, class-specific noise distributions might offer further refinement.
  • Assumption of GT accuracy: The study relies on the assumption that the reannotated boxes in [17] accurately reflect the true ground truth. If these reannotations also contain noise, it could impact the accuracy of the noise estimation.
  • Future Work: The authors propose extending the noise-aware framework to define object-aware thresholds, suggesting that smaller objects might benefit from more relaxed IoU thresholds compared to larger ones. They also aim to explore the framework for OBBs and 3D object detection.

Summary updated

BibTeX
@inproceedings{llerena2025noise,
  author = {Jeffri Murrugarra Llerena and Claudio R. Jung},
  title = {{Noise-Aware Evaluation of Object Detectors}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure