Inferring Spatial Uncertainty in Object Detection

Zining Wang, Di Feng, Yiyang Zhou, Lars Rosenbaum, Fabian Timm, Klaus Dietmayer, Masayoshi Tomizuka, Wei Zhan

TL;DR

This paper introduces a generative framework to estimate the inherent "label uncertainty" in 3D object detection datasets caused by sensor noise and occlusions. By representing bounding boxes as spatial probability distributions, the authors propose Jaccard IoU (JIoU), a new metric that allows for a more nuanced evaluation of probabilistic object detectors compared to standard deterministic metrics.

What Problem Does This Paper Address?

In autonomous driving, we rely on datasets like KITTI and Waymo to train object detectors. These datasets provide "ground truth" bounding boxes as deterministic boxes. However, these labels are rarely perfect. Human annotators often have to guess the full dimensions of a car based on limited "L-shaped" LiDAR reflections (where only the front and one side are visible).

Current evaluation metrics, like Intersection over Union (IoU) and Average Precision (AP), treat these labels as absolute truth. This creates two issues:

  1. Training Noise: Models are forced to fit noisy or ambiguous labels equally, which can hurt generalization.
  2. Evaluation Gap: Probabilistic detectors (which output a distribution rather than a single box) cannot be fairly evaluated against deterministic labels because the labels themselves lack an uncertainty measure.

Methods

The authors propose a three-part solution:

  1. Generative Label Uncertainty Model: Instead of treating a label as a fixed box, they use a generative model of LiDAR point clouds. Using Variational Bayes, they infer a posterior distribution of the bounding box parameters ($c_x, c_y, l, w$, etc.) given the observed LiDAR points. This captures the reality that the "back" of a car (where there are no points) is more uncertain than the "front" (where points are dense).
  2. Spatial Distribution ($p_G$): They define a new representation for probabilistic bounding boxes. Instead of just a mean and variance of parameters, they create a spatial probability density function (PDF) that describes the likelihood of any specific point in 3D space being part of the object.
  3. Jaccard IoU (JIoU): They extend the classic IoU metric to the probabilistic domain using the Jaccard index of probability distributions. JIoU measures the overlap between the predicted spatial distribution and the ground truth spatial distribution.

Datasets

The researchers validated their approach on two industry-standard benchmarks:

  • KITTI Dataset: Used for initial modeling and evaluation of "Car" objects (approx. 30k objects).
  • Waymo Open Dataset: Used to demonstrate scalability and performance across different ranges (up to 70m).

Key Insights and Ideas

  • The "L-Shape" Intuition: LiDAR typically only sees the surfaces of objects facing the sensor. The model correctly identifies that the corner closest to the sensor is the most "certain" part of the box, while the far corners have much higher variance.
  • Corners over Centers: A fascinating finding is that object detectors are actually better at predicting the nearest corner of an object than its center. By re-aligning predicted boxes based on corners rather than centers, the authors saw an immediate boost in Average Precision (AP) across multiple state-of-the-art architectures (like PointRCNN and VoxelNet).
  • JIoU as a Calibration Tool: JIoU penalizes "over-confident" models. If a model predicts a very tight, certain box where the ground truth label is actually highly uncertain, the JIoU score will be lower than standard IoU, encouraging models to produce better-calibrated uncertainty estimates.

Key Results

  • Improved Evaluation: Experiments showed that JIoU is superior to IoU when evaluating probabilistic detectors because it accounts for the "wellness" of the label.
  • Quantifying Label Noise: The authors used their model to identify "bad" labels in the KITTI dataset. Their method outperformed existing heuristics (like convex hull methods) in detecting labels that were likely erroneous or highly ambiguous.
  • Architecture Sensitivity: The study revealed that different architectures handle spatial uncertainty differently. For example, VoxelNet showed a significant $+8.0\%$ improvement in "Hard" category AP when using corner-alignment, highlighting how much localization error is actually just "center-guessing" error.

Limitations and Future Work

  • Object Diversity: The current generative model is optimized for vehicles (boxes). The authors note that more complex rendering models would be needed for pedestrians or cyclists, which have non-rigid shapes.
  • Training Integration: While this paper focuses on evaluation and inference, the authors suggest that incorporating these uncertain labels directly into the loss functions during training is a promising next step to build more robust detectors.
  • Computational Complexity: Calculating JIoU involves sampling points from distributions, which is more computationally expensive ($O(N \log N)$) than standard geometric IoU.

Summary updated

Abstract

The availability of real-world datasets is the prerequisite for developing object detection methods for autonomous driving. While ambiguity exists in object labels due to error-prone annotation process or sensor observation noises, current object detection datasets only provide deterministic annotations without considering their uncertainty. This precludes an in-depth evaluation among different object detection methods, especially for those that explicitly model predictive probability. In this work, we propose a generative model to estimate bounding box label uncertainties from LiDAR point clouds, and define a new representation of the probabilistic bounding box through spatial distribution. Comprehensive experiments show that the proposed model represents uncertainties commonly seen in driving scenarios. Based on the spatial distribution, we further propose an extension of IoU, called the Jaccard IoU (JIoU), as a new evaluation metric that incorporates label uncertainty. Experiments on the KITTI and the Waymo Open Datasets show that JIoU is superior to IoU when evaluating probabilistic object detectors.
BibTeX
@inproceedings{wang2020inferring,
  author = {Zining Wang and Di Feng and Yiyang Zhou and Lars Rosenbaum and Fabian Timm and Klaus Dietmayer and Masayoshi Tomizuka and Wei Zhan},
  title = {{Inferring Spatial Uncertainty in Object Detection}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=gLC8mLY9Dkk},
}

Loading related content...

Figure