NVAutoNet: Fast and Accurate 360deg 3D Visual Perception for Self Driving

Trung Pham, Mehran Maghoumi, Wanli Jiang, Bala Siva Sashank Jujjavarapu, Mehdi Sajjadi, Xin Liu, Hsuan-Chu Lin, Bor-Jeng Chen, Giang Truong, Chao Fang, Junghyun Kwon, Minwoo ParkWACV 2024

TL;DR

NVAutoNet is a novel 3D visual perception system for autonomous vehicles that achieves high accuracy and real-time performance by efficiently fusing multi-camera inputs into a Bird's-Eye-View (BEV) representation. It addresses critical challenges in scalability, sensor variability, and computational efficiency, making it suitable for deployment on embedded hardware like the NVIDIA DRIVE Orin SoC.

What Problem Does This Paper Address?

Autonomous vehicles rely heavily on accurate 3D perception to navigate safely. Traditional approaches often process individual camera feeds separately and then merge the results, which can lead to compounding errors and unknown error models. Furthermore, existing methods frequently prioritize detection accuracy over crucial factors like computational efficiency, ease of deployment on onboard hardware, resilience to sensor misalignments (due to different vehicle models), and adaptability to various vehicle types. This paper tackles these limitations by proposing NVAutoNet, a unified system designed for robust, real-time 3D perception that is also practical for automotive deployment.

Methods

NVAutoNet employs a multi-stage approach:

  • 2D Image Feature Extraction: Lightweight, hardware-aware CNN backbones are used to extract features from synchronized 360-degree camera inputs. These networks are optimized for speed using TensorRT and are configured using hardware-informed neural architecture search.
  • Image-to-BEV Transformation and Fusion: This is a core innovation. Instead of relying on traditional homography or depth-based methods, NVAutoNet uses a learned, MLP-based approach to transform 2D image features into a unified BEV feature map. This transformation leverages camera intrinsic and extrinsic parameters and uses a novel method of fitting polynomial curves to image columns to infer BEV positions, avoiding assumptions about flat worlds or rectified images. Multi-camera features are fused at the BEV level.
  • BEV Feature Extraction: A dedicated BEV encoder further processes the fused BEV features to extract higher-level representations.
  • Task-Specific Heads: The final BEV features are fed into task-specific heads to predict 3D signals, including obstacles, freespaces, and parking spaces. Crucially, these tasks are structured as set prediction problems, eliminating the need for post-processing steps like Non-Maximum Suppression (NMS) or clustering.

Datasets

The paper primarily uses an extensive, proprietary in-house dataset comprising real, simulated, and augmented reality data. This dataset includes 2.2 million training scenes, 400K validation scenes, and 177K testing scenes, collected from over 20 countries under diverse conditions. The authors note that many existing public datasets (like nuScenes) are less suitable for their work due to their focus on shorter detection ranges (under 70 meters) compared to NVAutoNet's target of 200 meters.

Key Insights and Ideas

  • BEV as a Unified Representation: The paper strongly advocates for the Bird's-Eye-View (BEV) as a superior representation for fusing multi-camera data, simplifying downstream tasks like prediction and planning.
  • Learned 2D-to-BEV Transformation: Moving beyond traditional geometric methods, NVAutoNet's learned transformation using MLPs and polynomial fitting offers a more robust and adaptable way to project image features into BEV space, especially for wide-angle or unrectified cameras.
  • Efficiency through Set Prediction: Structuring perception tasks as set prediction problems significantly streamlines the pipeline by removing computationally expensive post-processing steps.
  • Hardware-Aware Optimization: The explicit focus on optimizing CNN backbones for TensorRT and embedded hardware (NVIDIA DRIVE Orin) is key to achieving real-time performance.
  • Adaptive Loss Balancing: The proposed adaptive weight adjustment for multi-task learning helps balance the contributions of different tasks, preventing one task from dominating the training process.

Key Results

  • Real-time Performance: NVAutoNet achieves an impressive 53 frames per second (FPS) on the NVIDIA DRIVE Orin SoC, with a total latency of just 18.7 ms per input.
  • High Accuracy: The system demonstrates strong performance across multiple tasks. For 3D obstacle detection, it achieves an overall mAP of 0.465, with particularly high AP for vehicles (0.648). For parking space detection, the mean IoU for true positives is around 86%.
  • Resilience and Adaptability: The system shows resilience to sensor mounting deviations and can be adapted to different vehicle platforms (e.g., car to truck) with minimal fine-tuning.
  • Speedup: Compared to BEVDET, NVAutoNet offers a 7.5x speed boost, and it outperforms Fast-BEV (45 FPS vs. 53 FPS) on the same hardware.

Limitations and Future Work

  • VRU Detection: The paper notes that while obstacle detection is strong, detecting persons/pedestrians is more challenging, likely due to their smaller size and typical size discrepancies compared to vehicles.
  • Parallel Parking: The parallel parking space category shows lower performance, attributed to labeling intricacies and variability in parallel parking spaces.
  • Future Work: The authors suggest that NVAutoNet can be extended to address online mapping perception tasks. The architecture's flexibility also lends itself to further exploration in adapting to new sensor configurations and improving performance on challenging categories like VRUs.

Summary updated

BibTeX
@inproceedings{pham2024nvautonet,
  author = {Trung Pham and Mehran Maghoumi and Wanli Jiang and Bala Siva Sashank Jujjavarapu and Mehdi Sajjadi and Xin Liu and Hsuan-Chu Lin and Bor-Jeng Chen and Giang Truong and Chao Fang and Junghyun Kwon and Minwoo Park},
  title = {{NVAutoNet: Fast and Accurate 360deg 3D Visual Perception for Self Driving}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure