TORNADO-Net: mulTiview tOtal vaRiatioN semAntic segmentation with Diamond inceptiOn module

Martin Gerdzhev, Ryan Razani, Ehsan Taghavi, Bingbing LiuICRA 2021 · DOI

TL;DR

This paper introduces TORNADO-Net, a novel deep learning architecture for 3D LiDAR point cloud semantic segmentation. It achieves state-of-the-art results on the SemanticKITTI benchmark by effectively fusing multi-view features and introducing a new Total Variation (TV) loss function to improve prediction smoothness.

What Problem Does This Paper Address?

Autonomous driving and robotics heavily rely on understanding the 3D environment. A critical component of this is semantic segmentation of LiDAR point clouds, which involves assigning a class label (e.g., road, car, pedestrian) to each point. LiDAR data presents unique challenges: it's unstructured, sparse, and point density varies with distance. While image-based segmentation is mature, adapting these techniques to 3D point clouds is difficult. Existing methods often project point clouds into structured representations like Bird's Eye View (BEV) or range images, but these projections can lose information or struggle with local context. Furthermore, class imbalance in datasets and the need for smooth, accurate predictions remain significant challenges.

Methods

TORNADO-Net tackles these issues with a multi-pronged approach:

  • Pillar Projection Learning (PPL): This module processes raw point clouds by grouping points into "pillars" and extracting features in a Bird's Eye View (BEV) representation. This helps capture local geometric information. The extracted features are then projected onto a range image for further processing.
  • Diamond Contextual Block (DCB): A novel module designed to capture global context. It uses a series of 2D convolutions with increasing kernel sizes (3x3, 5x5, 7x7) in a "diamond" shape, along with skip connections, to extract rich contextual features from the range image.
  • Encoder-Decoder Architecture: A standard encoder-decoder network, similar to SalsaNet, is used for feature extraction and upsampling. However, it incorporates dilated convolutions for a larger receptive field without increasing parameters.
  • Novel Loss Function: The paper proposes a new loss function that combines:
  • Weighted Cross-Entropy (WCE): Addresses class imbalance.
  • Lovász-Softmax Loss: Directly optimizes the Intersection over Union (IoU) metric, which is crucial for segmentation performance.
  • Total Variation (TV) Loss: A key innovation, this loss regularizes predictions by encouraging smoothness over neighboring points, effectively denoising and reducing noisy outputs. This is inspired by image denoising techniques.
  • Circular Padding: Used to handle the 360° field of view of LiDAR sensors, preventing edge artifacts.

Datasets

The primary dataset used for training and evaluation is SemanticKITTI. This dataset provides dense, point-wise semantic annotations for LiDAR sequences, making it a standard benchmark for 3D semantic segmentation. The authors use the training and validation split for experiments and ablation studies, with sequence 08 specifically used for ablation.

Key Insights and Ideas

  • Multi-view Fusion is Key: Combining features from different projections (BEV and range image) is crucial for capturing a comprehensive understanding of the scene, as different views highlight different aspects of the geometry.
  • Local Context Matters for Smoothness: Standard segmentation losses often overlook the relationship between neighboring points, leading to noisy predictions. The proposed TV loss effectively leverages this local neighborhood information to produce smoother, more accurate segmentations.
  • Global Context for Richer Features: The Diamond Contextual Block demonstrates that specialized convolutional structures can extract more meaningful global context from range images, improving overall segmentation quality.
  • Loss Function Synergy: The combination of WCE, Lovász-Softmax, and the novel TV loss provides a powerful training objective that balances class balance, IoU optimization, and prediction smoothness.

Key Results

TORNADO-Net achieves state-of-the-art results on the SemanticKITTI benchmark, outperforming previous methods in mean IoU (mIoU) across most classes.

  • The proposed method achieves a mIoU of 61.2% (low-res) and 61.8% (hi-res) on the SemanticKITTI test split, surpassing previous methods.
  • The TV Loss contribution is highlighted as particularly impactful, leading to the biggest improvements in mIoU.
  • The PPL module provides a significant jump in mIoU (around 1.5%).
  • Circular padding offers a modest but positive effect (around 0.5%).
  • KNN post-processing further boosts performance by 2-3%.
  • The model is also competitive in terms of Frames Per Second (FPS), making it applicable to real-time systems, outperforming many point-based methods in speed.

Qualitative results show TORNADO-Net accurately segmenting various objects and scenes, with some challenging cases (like distinguishing between similar vehicle classes) highlighted as areas for future improvement.

Limitations and Future Work

  • Distinguishing Similar Classes: The paper acknowledges that distinguishing between very similar classes (e.g., car vs. truck) can still be challenging, likely due to limited training data and shared features. This is identified as an area for future research.
  • Computational Cost: While faster than many point-based methods, using a hi-res version and KNN post-processing increases computational cost and reduces speed, which might be a trade-off for real-time applications.
  • Future Work: The authors suggest further research into addressing the challenges of distinguishing similar classes. The paper also implies that the proposed architecture and loss functions could be adapted for other 3D perception tasks.

Summary updated

BibTeX
@inproceedings{gerdzhev2021tornado,
  author = {Martin Gerdzhev and Ryan Razani and Ehsan Taghavi and Bingbing Liu},
  title = {{TORNADO-Net: mulTiview tOtal vaRiatioN semAntic segmentation with Diamond inceptiOn module}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9562041},
}
Figure