Human Vision Based 3D Point Cloud Semantic Segmentation of Large-Scale Outdoor Scenes

Sunghwan Yoo, Yeonjeong Jeong, Maryam Jameela, Gunho SohnWorkshops @ CVPR 2023 · DOI · arXiv

Introduction: The Challenge of Large-Scale 3D Mapping

Autonomous driving and urban planning rely heavily on LiDAR-generated 3D point clouds. Segmenting these massive datasets—assigning semantic labels like "road," "building," or "vegetation" to billions of points—presents a significant computational challenge. Researchers typically face a trade-off between data density and scene coverage. High-density crops preserve fine details but lack global context, while downsampling to save memory loses the geometric precision needed for small object recognition.

EyeNet: Mimicking Human Vision

To address this, the authors propose EyeNet, a novel architecture inspired by the human eye's foveal and peripheral vision. Instead of processing a uniform area, EyeNet utilizes a multi-scale sampling strategy that captures a high-density "Central Region" for fine details and a larger, low-density "Peripheral Region" for environmental context.

The EyeNet architecture processes central and peripheral regions in parallel streams, connected by specialized blocks to exchange information.

Human-Inspired Sampling

The input is divided into three distinct point sets:

  • Central Region: A high-density sphere focused on the target area.
  • Peripheral Region: A larger, sparse area providing broad context.
  • Messenger Points: A subset of points shared between both regions, facilitating information flow.

Parallel Processing and Connection Blocks

EyeNet employs two parallel streams based on the RandLA-Net backbone. The "secret sauce" lies in the Connection Blocks, which use self-attentive pooling and channel enhancement to allow the streams to communicate via messenger points. This ensures that local geometric features are informed by the surrounding global context. Finally, a Feature Merging Block with squeeze-and-excitation modules combines the streams for the final prediction.

Key Insights

  • Efficiency: By using a sparse periphery, EyeNet covers the same spatial area as standard models while using 60% fewer points. This allows for significantly larger batch sizes during inference.
  • Contextual Advantage: The peripheral vision approach is particularly effective for large-scale structures like bridges and rail lines that require a wide receptive field.
  • Information Exchange: Ablation studies confirm that the explicit feature exchange in the Connection Blocks is vital, contributing nearly 1% to the overall Mean Intersection over Union (mIoU).

Key Results

EyeNet was evaluated on major benchmarks including SensatUrban, Toronto3D, and DALES. It consistently outperformed existing state-of-the-art methods.

Quantitative comparison showing EyeNet outperforming RandLA and KPConv across various semantic classes.

On the SensatUrban dataset, EyeNet achieved an mIoU of 62.30%, a 5% improvement over RandLA++. On Toronto3D, it reached 81.13% mIoU, showing particular strength in identifying difficult classes like road markings. Visual analysis showed that EyeNet produces much cleaner segmentations for large buildings compared to fragmented results from baseline models.

Conclusion and Future Work

EyeNet demonstrates that human-vision-inspired sampling can significantly improve the efficiency and accuracy of 3D semantic segmentation. While currently implemented with a RandLA-Net backbone, the approach is model-agnostic. Future work will explore integrating this strategy with more advanced backbones like Point Transformers and refining performance on extremely sparse aerial datasets.

Summary updated

BibTeX
@inproceedings{yoo2023human,
  author = {Sunghwan Yoo and Yeonjeong Jeong and Maryam Jameela and Gunho Sohn},
  title = {{Human Vision Based 3D Point Cloud Semantic Segmentation of Large-Scale Outdoor Scenes}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00699},
}
Figure