VIN: Voxel-based Implicit Network for Joint 3D Object Detection and Segmentation for Lidars

Yuanxin Zhong, Minghan Zhu, Huei Peng

In the world of autonomous driving, "seeing" is a multi-layered challenge. A self-driving car doesn't just need to know where the other cars are (3D Object Detection); it needs to understand the entire geometry of the world, from the drivable surface to the sidewalk vegetation (Semantic Segmentation). Historically, these have been treated as separate problems, often requiring two different neural networks running in parallel—a luxury that real-time embedded systems can rarely afford.

In their paper, "VIN: Voxel-based Implicit Network for Joint 3D Object Detection and Segmentation for Lidars," researchers from the University of Michigan present a unified architecture that tackles both tasks simultaneously without breaking the computational budget.

TL;DR

VIN (Voxel-based Implicit Network) is a unified framework that performs 3D object detection and point cloud segmentation in a single pass. By adding a lightweight "implicit function" branch to a standard voxel-based detector, it achieves state-of-the-art segmentation results on the nuScenes dataset with almost zero additional latency and works remarkably well even with only 0.1% of semantic labels.

What Problem Does This Paper Address?

Lidar-based perception usually splits into two camps: detection (drawing 3D boxes around "things" like cars and pedestrians) and segmentation (labeling every single point as "stuff" like road, sidewalk, or grass).

The authors identify three major pain points in current approaches:

  1. Efficiency: Running separate models for detection and segmentation is computationally expensive.
  2. Precision Loss: Voxel-based methods (which group points into a 3D grid) are fast but suffer from "rasterization error"—the loss of fine-grained detail when points are squashed into a grid.
  3. Label Scarcity: Point-wise semantic labels are incredibly tedious and expensive to produce compared to 3D bounding boxes.

Methods

The core of the paper is the VIN architecture, which builds upon a standard voxel-based detector (specifically, CenterPoint).

The VIN architecture featuring the implicit function branch and InConsistency Suppression (ICS).
  • The Implicit Rendering Branch: This is the "secret sauce." Instead of predicting a label for a whole voxel, the network learns an implicit function—a small Multi-Layer Perceptron (MLP). This function takes a specific point's coordinates and its local voxel features to predict a semantic label. Because it's a continuous function, it can predict labels for any coordinate, effectively "rendering" the semantics and recovering the precision lost during voxelization.
  • InConsistency Suppression (ICS): The authors noticed that detection and segmentation don't always agree (e.g., a point labeled "sidewalk" might be inside a "car" bounding box). They developed a post-processing algorithm (ICS) that uses the confidence scores of both branches to "vote" on the correct label, fixing inconsistencies in-place.
  • Multi-Task Learning: The model is trained using a combined loss function that supervises both the bounding box parameters and the point-wise semantic distributions.

Datasets

The researchers evaluated VIN on the nuScenes-lidarseg dataset. This is a massive, real-world outdoor dataset containing 1,000 scenes with 3D bounding boxes and point-wise semantic labels for 15 different categories. It is particularly challenging due to diverse weather conditions and night-time driving scenarios.

Key Insights and Ideas

  • Implicit is Better: By using an implicit representation, the model decouples feature extraction from the final prediction. This allows the network to maintain high-resolution semantic outputs even when the underlying feature map is relatively coarse.
  • Detection as Supervision: One of the most striking findings is that the detection task actually helps the segmentation task. The bounding boxes provide a "strong hint" about what the points inside should be, which is why the model performs so well under weak supervision.
  • Efficiency is Free: The semantic branch is so lightweight (a 4-layer MLP) that the total inference speed only drops from 6.0 FPS to 5.9 FPS compared to the base detector. This makes it a "near-free" upgrade for existing detection pipelines.

Key Results

VIN demonstrates significant improvements over existing benchmarks, particularly in semantic segmentation tasks where it sets a new state-of-the-art for the nuScenes dataset.

Performance comparison on the nuScenes-lidarseg dataset.
  • Segmentation Performance: VIN achieved 73.8% mIoU (mean Intersection over Union) on nuScenes, outperforming previous state-of-the-art methods like SalsaNext and MinkNet42 by roughly 9.4 percentage points. As shown in the table, it excels in categories like "Car" (87.0) and "Drivable Area" (94.5).
  • Weak Supervision: In a "weakly supervised" test, the model was trained with only 0.1% of the semantic labels. Remarkably, it still achieved a 72.3% mIoU—nearly on par with the model trained on 100% of the labels.
  • Detection Quality: While there was a slight drop in detection accuracy (mAP) compared to a standalone CenterPoint model (46.3 vs 50.7), the results remain highly competitive while providing the added benefit of full-scene segmentation.

Limitations and Future Work

Despite its strengths, VIN has room to grow:

  • Small/Sparse Objects: Like most Lidar-only methods, VIN struggles with very small objects or those with very few Lidar reflections (e.g., distant pedestrians or thin poles).
  • "Stuff" vs. "Things": The authors suggest that future iterations could include explicit penalty terms to prevent "stuff" (like road) from being predicted inside "thing" boxes (like cars), further refining the ICS algorithm.
  • Sensor Fusion: The authors acknowledge that Lidar has inherent limits in dark or dazzling conditions and suggest that incorporating camera data (multi-modal fusion) is the logical next step to improve robustness.

Summary updated

BibTeX
@inproceedings{zhong2021vin,
  author = {Yuanxin Zhong and Minghan Zhu and Huei Peng},
  title = {{VIN: Voxel-based Implicit Network for Joint 3D Object Detection and Segmentation for Lidars}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=iVAW8g4VZl},
}
Figure