Abstract
Multi-modality fusion is proven an effective method for 3d perception for autonomous driving. However, most current multi-modality fusion pipelines for LiDAR semantic segmentation have complicated fusion mechanisms. Point painting is a quite straight forward method which directly bind LiDAR points with visual information. Unfortunately, previous point painting like methods suffer from projection error between camera and LiDAR. In our experiments, we find that this projection error is the devil in point painting. As a result of that, we propose a depth aware point painting mechanism, which significantly boosts the multi-modality fusion. Apart from that, we take a deeper look at the desired visual feature for LiDAR to operate semantic segmentation. By Lifting Visual Information as Cue, LVIC ranks 1st on nuScenes LiDAR semantic segmentation benchmark. Our experiments show the robustness and effectiveness. Codes would be make publicly available soon.
Introduction
In autonomous driving, LiDAR provides precise 3D geometry, while cameras provide rich texture and color. Combining them (multi-modality fusion) is a logical step for better scene understanding. However, current fusion pipelines are often overly complex or rely on "point painting"—a method where LiDAR points are simply assigned the color or features of the pixel they project onto.
The authors identify a "devil" in point painting: projection error. Even with good calibration, LiDAR and cameras are physically separated, leading to parallax and misalignment, especially for distant objects. Naive models can't distinguish between a correct projection and a "mis-painted" one, leading to noisy data that can degrade segmentation performance.
Methods
The LVIC framework introduces three main components to solve the fusion problem efficiently:
- Visual Encoder (EfficientViT): Instead of using high-level semantic features, the authors use an EfficientViT-b0 backbone to extract low-level appearance features. They argue these raw textures are more useful for augmenting LiDAR points. Crucially, this encoder also produces a monocular depth map.
- Depth-Aware Painting Module: For every LiDAR point, the system projects it onto the camera image and appends the 2D pixel coordinates, the vision-estimated depth ($z_c$), and the texture feature vector.
- Fusion Module: A "plug-and-play" module uses simple linear layers to adapt the visual features for the LiDAR domain. By having both the true LiDAR depth and the camera-estimated depth, the network can learn to use the difference between the two as a "confidence score." If the depths don't match, the network learns to ignore the visual "paint."
Key Insights
- Depth as a Filter: The most significant innovation is using monocular depth estimation as a validation tool. If the camera thinks a pixel is 10 meters away, but the LiDAR point is at 50 meters, the projection is clearly wrong. LVIC lets the model learn this relationship automatically.
- Low-Level Features Win: The authors found that low-level texture features are more robust for LiDAR painting than high-level object embeddings. This suggests that the LiDAR backbone is already good at "semantics," and what it really needs from the camera is raw visual "context."
Key Results
LVIC achieved 1st place on the nuScenes LiDAR semantic segmentation leaderboard at the time of the report's release.
- Overall Performance: Reached a Mean Intersection over Union (mIOU) of 83.8, compared to the baseline (Pep) at 81.8.
- Category Gains: The method showed dramatic improvements in challenging categories, such as Bicycle (from 55.5 to 68.7 mIOU) and Construction Vehicle (from 72.7 to 81.5 mIOU).
Conclusion
LVIC demonstrates that addressing projection error through depth-aware painting significantly boosts multi-modality fusion performance. By simplifying the fusion mechanism and focusing on low-level visual cues, the model achieves state-of-the-art results on the nuScenes benchmark while remaining efficient. Future work will explore extending this logic to other 3D perception tasks.
Summary updated