SeaBird: Segmentation in Bird's View with Dice Loss Improves Monocular 3D Detection of Large Objects

Abhinav Kumar, Yuliang Guo, Xinyu Huang, Liu Ren, Xiaoming Liu

Abstract

Monocular 3D detectors achieve remarkable performance on cars and smaller objects. However, their performance drops on larger objects, leading to fatal accidents. Some attribute the failures to training data scarcity or the receptive field requirements of large objects. In this paper, we highlight this understudied problem of generalization to large objects. We find that modern frontal detectors struggle to generalize to large objects even on nearly balanced datasets. We argue that the cause of failure is the sensitivity of depth regression losses to noise of larger objects. To bridge this gap, we comprehensively investigate regression and dice losses, examining their robustness under varying error levels and object sizes. We mathematically prove that the dice loss leads to superior noise-robustness and model convergence for large objects compared to regression losses for a simplified case. Leveraging our theoretical insights, we propose SeaBird (Segmentation in Bird's View) as the first step towards generalizing to large objects. SeaBird effectively integrates BEV segmentation on foreground objects for 3D detection, with the segmentation head trained with the dice loss. SeaBird achieves SoTA results on the KITTI-360 leaderboard and improves existing detectors on the nuScenes leaderboard, particularly for large objects.

Introduction: The Large Object Challenge

Monocular 3D object detection (Mono3D) is a critical task for autonomous vehicles, requiring the estimation of 3D bounding boxes from single 2D images. While current models have reached high accuracy for standard cars, they often fail catastrophically when encountering large objects like buses, trucks, or trailers.

Previously, these failures were attributed to data scarcity or limited receptive fields. However, the authors of SeaBird discovered that even on balanced datasets, standard "frontal" detectors still struggle. They identify a more fundamental culprit: depth regression losses (like L1 or L2). These losses are highly sensitive to noise in depth prediction, and this sensitivity scales poorly as object size increases. When a model tries to regress the exact depth of a massive trailer, small errors in pixel-level depth lead to massive variances in the loss, preventing the model from converging to an optimal solution.

The SeaBird Architecture

To address the limitations of regression losses, the authors propose SeaBird (Segmentation in Bird’s View), a two-stage pipeline designed to harness the noise-robustness of segmentation losses.

The SeaBird pipeline integrates a BEV segmentation head trained with Dice loss to guide the final Mono3D detection head.

Methods

  1. BEV Segmentation with Dice Loss: Instead of directly regressing depth, SeaBird transforms image features into a Bird’s Eye View (BEV). It then uses a segmentation head to predict foreground object masks. Crucially, this head is trained using Dice loss.
  2. Sequential Integration: Unlike models that run detection and segmentation in parallel, SeaBird is sequential. The predicted BEV segmentation map is concatenated with the original BEV features. This refined feature stack is then passed to a Mono3D detection head to predict the final 7-DoF boxes (position, elevation, dimensions, and yaw).
  3. Two-Stage Training: The model first trains the BEV segmentation head to exploit its superior convergence properties. In the second stage, the detection head is fine-tuned using the segmentation output as a reliable spatial guide.

Theoretical Foundation: Why Dice Loss?

The core innovation is supported by a mathematical proof (Theorem 1) showing that the gradient variance of Dice loss is significantly lower than that of L1/L2 losses when noise is present, particularly for objects with a larger spatial footprint. As object length ($l$) increases, the convergence gap between Dice loss and regression losses widens. Essentially, the larger the object, the more "help" it gets from the Dice loss's inherent noise-robustness.

Experimental Results

The researchers evaluated SeaBird on two major benchmarks: KITTI-360, which contains a near 1:2 ratio of large objects to cars, and nuScenes, a massive multi-camera dataset.

SeaBird achieves State-of-the-Art results on KITTI-360 and nuScenes, while theoretical analysis confirms the superior noise robustness of Dice loss (L_dice) over regression losses (L1, L2).

Key Findings

  • SoTA on KITTI-360: SeaBird achieved State-of-the-Art results on the KITTI-360 leaderboard, outperforming all previous monocular methods and even some older LiDAR-based baselines.
  • Boosting Existing Models: When integrated into the HoP detector on the nuScenes dataset, SeaBird improved the detection of large objects ($AP_{Lrg}$) by 3.8% and overall NDS by 1.9%.
  • Noise Robustness: Empirical tests confirmed that SeaBird maintains high accuracy even as depth noise increases, whereas standard regression-based models see their performance plummet as noise standard deviation ($\sigma$) rises.

Conclusion and Future Work

SeaBird demonstrates that shifting from depth regression to BEV segmentation with Dice loss provides a more stable and robust path for detecting large objects. While the current theoretical proof relies on an idealized model with Gaussian noise, empirical results suggest these findings hold true in complex deep learning environments.

Future work may explore more generalized mathematical frameworks and investigate how to effectively incorporate environmental context (background categories) without distracting the network from critical foreground objects, as current experiments showed that including background segmentation actually decreased performance.

Summary updated

Abstract

Monocular 3D detectors achieve remarkable performance on cars and smaller objects. However, their performance drops on larger objects, leading to fatal accidents. Some attribute the failures to training data scarcity or the receptive field requirements of large objects. In this paper, we highlight this understudied problem of generalization to large objects. We find that modern frontal detectors struggle to generalize to large objects even on nearly balanced datasets. We argue that the cause of failure is the sensitivity of depth regression losses to noise of larger objects. To bridge this gap, we comprehensively investigate regression and dice losses, examining their robustness under varying error levels and object sizes. We mathematically prove that the dice loss leads to superior noise-robustness and model convergence for large objects compared to regression losses for a simplified case. Leveraging our theoretical insights, we propose SeaBird (Segmentation in Bird's View) as the first step towards generalizing to large objects. SeaBird effectively integrates BEV segmentation on foreground objects for 3D detection, with the segmentation head trained with the dice loss. SeaBird achieves SoTA results on the KITTI-360 leaderboard and improves existing detectors on the nuScenes leaderboard, particularly for large objects.
BibTeX
@misc{kumar2024seabird,
  author = {Abhinav Kumar and Yuliang Guo and Xinyu Huang and Liu Ren and Xiaoming Liu},
  title = {{SeaBird: Segmentation in Bird's View with Dice Loss Improves Monocular 3D Detection of Large Objects}},
  year = {2024},
  eprint = {2403.20318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.20318},
}
Figure