Alleviating Foreground Sparsity for Semi-Supervised Monocular 3D Object Detection

Weijia Zhang, Dongnan Liu, Chao Ma, Tom Weidong Cai

TL;DR

This paper introduces ODM3D, a semi-supervised monocular 3D object detection framework that leverages LiDAR data to improve performance. It tackles the challenge of "foreground sparsity" in training data by using LiDAR point clouds to guide knowledge distillation and a novel data augmentation strategy, achieving state-of-the-art results on the KITTI benchmark.

What Problem Does This Paper Address?

Monocular 3D object detection (M3OD) aims to identify and locate objects in 3D space using only a single RGB camera. This is a highly desirable capability for autonomous driving due to its low cost and passive sensing nature. However, it's a notoriously difficult task because RGB images lack explicit depth information, making it an "ill-posed problem." Existing methods often struggle with performance gaps compared to LiDAR-based detectors. A key challenge identified by the authors is "foreground sparsity," where training datasets, especially those with unlabelled data, often contain many scenes with few or no objects of interest. This leads to insufficient training signals and can cause foreground object features to be overwhelmed by background noise during training, particularly in Bird's-Eye View (BEV) representations.

Methods

The proposed ODM3D framework builds upon a teacher-student paradigm with semi-supervised learning and cross-modal knowledge distillation.

  • Cross-Modal Knowledge Distillation: A pre-trained LiDAR-based 3D object detector (the "teacher") is used to guide a monocular detector (the "student"). Knowledge is transferred in both the feature and prediction spaces, specifically in the BEV representation.
  • BEV Occupancy Guidance Mask: To address foreground sparsity and ensure distillation focuses on relevant areas, the authors introduce a novel "BEV occupancy mask." This mask is derived from the LiDAR point cloud and indicates which grid cells in the BEV space are likely to contain objects. By imposing this mask during distillation, the framework prioritizes learning from foreground regions and ignores background noise.
  • Occupancy-Guided Feature and Response Distillation: The BEV occupancy mask is applied to guide both the distillation of intermediate BEV features and the final detection predictions (classification, localization, and direction). Gaussian smoothing is applied to the occupancy mask to mitigate potential misalignment issues.
  • Occlusion-Aware Cross-Modal Data Augmentation (CMAug): To further combat foreground sparsity and improve RGB-LiDAR joint learning, a new data augmentation strategy is proposed. It addresses limitations of existing methods that use Intersection over Union (IoU) for collision tests. The authors introduce the Occlusion-Aware Intersection Score (OAIS), which considers the relative depth of objects to better identify severe occlusions. This prevents the augmentation from introducing misleading training samples. Additionally, a filter is introduced to discard augmented objects that are excessively small in the image's perspective view (PV).

Datasets

  • KITTI 3D Dataset: Used for both training and evaluation. This dataset includes 7,481 training and 7,518 test images with corresponding point clouds. The training set is further split into validation and training subsets.
  • KITTI Raw Dataset: A larger collection of unannotated images and point clouds, exploited for semi-supervised learning.

Key Insights and Ideas

  • Foreground Sparsity is a Major Bottleneck: The paper clearly identifies and quantifies the negative impact of sparse object presence in training data on M3OD performance.
  • LiDAR Point Clouds as Implicit Guides: Leveraging the spatial distribution of LiDAR points in BEV provides a powerful, implicit signal for where objects are likely to be, enabling more focused and effective cross-modal learning.
  • BEV Occupancy is Key to Focused Distillation: Directly using BEV occupancy information to guide feature and response distillation significantly improves the efficiency and effectiveness of knowledge transfer from the LiDAR teacher to the monocular student.
  • IoU is Insufficient for Cross-Modal Augmentation: The standard IoU metric for collision detection in data augmentation fails to account for 3D occlusions, leading to problematic training samples. The proposed OAIS metric offers a more robust solution.
  • Augmentation Needs to be "Learner-Friendly": Simply pasting objects isn't enough; the augmentation must be designed to avoid introducing confusing or misleading signals, especially concerning occlusions and object scale in the image.

Key Results

  • State-of-the-Art Performance: ODM3D achieves the 1st place ranking on both KITTI validation and test benchmarks for monocular 3D object detection, outperforming all existing supervised and semi-supervised methods in both BEV and 3D detection metrics.
  • Significant Performance Boost: The framework significantly improves the baseline monocular detector (CaDDN) by 42.4% on moderate cars.
  • Outperforming CMKD: ODM3D surpasses the previous state-of-the-art semi-supervised method, CMKD, by notable margins (e.g., 0.40 AP3D on KITTI test).
  • Effective Component Contributions: Ablation studies demonstrate that both the occupancy-guided distillation and the CMAug strategy contribute significantly to the overall performance gains.
  • Qualitative Improvements: Visualizations show ODM3D better handles faraway and occluded objects compared to CMKD.

Limitations and Future Work

  • Inference Efficiency: The paper notes that the proposed methods are primarily for training and do not introduce additional computational or memory overhead at inference, which is a strong point. However, the teacher model is discarded at inference, implying the student model is the sole component used.
  • Generalization of CMAug: The paper demonstrates CMAug's effectiveness on other multi-modal detectors, suggesting its broader applicability beyond M3OD. This hints at future work exploring its integration with other tasks.
  • Further Exploration of Occupancy Guidance: While effective, the paper suggests that further exploration of occupancy guidance in different settings and tasks could be beneficial.

Summary updated

Abstract

Monocular 3D object detection (M3OD) is a significant yet inherently challenging task in autonomous driving due to absence of explicit depth cues in a single RGB image. In this paper, we strive to boost currently underperforming monocular 3D object detectors by leveraging an abundance of unlabelled data via semi-supervised learning. Our proposed ODM3D framework entails cross-modal knowledge distillation at various levels to inject LiDAR-domain knowledge into a monocular detector during training. By identifying foreground sparsity as a main culprit behind existing methods’ suboptimal training, we exploit the precise localisation information embedded in LiDAR points to enable more foreground-attentive and efficient distillation via the proposed BEV occupancy guidance mask, leading to notably improved knowledge transfer and M3OD performance. Besides, motivated by insights into why existing cross-modal GT-sampling techniques fail on our task at hand, we further design a novel cross-modal object-wise data augmentation strategy for effective RGB-LiDAR joint learning. Our method ranks 1 st in both KITTI validation and test benchmarks, significantly surpassing all existing monocular methods, supervised or semi-supervised, on both BEV and 3D detection metrics. Code will be released at https://github.com/arcaninez/odm3d.
BibTeX
@inproceedings{zhang2024alleviating,
  author = {Weijia Zhang and Dongnan Liu and Chao Ma and Tom Weidong Cai},
  title = {{Alleviating Foreground Sparsity for Semi-Supervised Monocular 3D Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00737},
}
Figure