DRINet: A Dual-Representation Iterative Learning Network for Point Cloud Segmentation

Maosheng Ye, Shuangjie Xu, Tongyi Cao, Qifeng Chen

Abstract

We present a novel and flexible architecture for point cloud segmentation with dual-representation iterative learning. In point cloud processing, different representations have their own pros and cons. Thus, finding suitable ways to represent point cloud data structure while keeping its own internal physical property such as permutation and scale-invariant is a fundamental problem. Therefore, we propose our work, DRINet, which serves as the basic network structure for dual-representation learning with great flexibility at feature transferring and less computation cost, especially for large-scale point clouds. DRINet mainly consists of two modules called Sparse Point-Voxel Feature Extraction and Sparse Voxel-Point Feature Extraction. By utilizing these two modules iteratively, features can be propagated between two different representations. We further propose a novel multi-scale pooling layer for pointwise locality learning to improve context information propagation. Our network achieves state-of-the-art results for point cloud classification and segmentation tasks on several datasets while maintaining high runtime efficiency. For large-scale outdoor scenarios, our method outperforms state-of-the-art methods with a real-time inference time of 62ms per frame.

Introduction

Processing 3D point clouds—the raw data generated by LiDAR sensors—is notoriously difficult because the data is sparse, irregular, and unordered. Researchers generally choose between two paths: point-based methods, which operate directly on raw points to preserve fine-grained details but struggle with computational efficiency in large scenes, and voxel-based methods, which discretize space into a 3D grid to use efficient convolutions but often lose local geometric precision.

While "dual-representation" models attempt to combine both, they typically use simple one-way fusion and rely on expensive operations like trilinear interpolation to move data between domains. DRINet addresses the need for a flexible, iterative architecture that can handle hundreds of thousands of points per frame without sacrificing accuracy or real-time speed.

Methodology

The core of DRINet is its Iterative Dual-Representation Learning loop, which allows point and voxel features to mutually refine each other through multiple passes.

Overview of the DRINet architecture showing the iterative SPVFE and SVPFE modules.

The architecture consists of three primary modules:

  • Geometry-Aware Feature Extraction (GAFE): Before the main loop, this module extracts initial features by looking at the relative positions of points within their voxels, ensuring the network remains aware of physical scale and local structure.
  • Sparse Point-Voxel Feature Extraction (SPVFE): This module converts point features into voxel features. To capture context without the heavy overhead of K-Nearest Neighbor (KNN) searches, the authors introduce a Multi-scale Pooling Layer. It uses simple MLPs and scattering operations across different voxel grid scales to aggregate local information efficiently.
  • Sparse Voxel-Point Feature Extraction (SVPFE): This module maps voxel features back to points. Instead of standard trilinear interpolation—which is slow and memory-intensive—DRINet uses a novel Attentive Gathering layer. This learnable layer uses local geometric priors (like offsets from the voxel center) to weight how voxel features are assigned to individual points.

Experimental Results

DRINet was evaluated across several standard benchmarks, including SemanticKITTI (outdoor LiDAR), S3DIS (indoor scans), ModelNet40 (classification), and ShapeNet (part segmentation).

Performance (mIoU) vs. Runtime (Hz) on the SemanticKITTI dataset. DRINet (labeled 'O') achieves superior accuracy at real-time speeds.

On the SemanticKITTI test set, DRINet achieved a state-of-the-art 67.5 mIoU, outperforming established models like RandLA-Net (53.9) and SPVNet (67.0). Crucially, the model maintains high efficiency, processing a frame in roughly 62ms (~15Hz) on an Nvidia RTX 2080 Ti. This real-time performance is achieved by eliminating the "KD-Tree" building process required for KNN and replacing static interpolation with the more efficient Attentive Gathering layer.

Ablation studies validated these architectural choices: removing the Multi-scale Pooling layer dropped performance by 1.9% mIoU, while removing Attentive Gathering resulted in a 0.8% decrease.

Conclusion

DRINet demonstrates that iterative feature propagation between point and voxel representations provides a significant boost in segmentation accuracy while maintaining the efficiency required for autonomous systems. By replacing computationally expensive geometric operations with learnable, scattering-based alternatives, the network sets a new standard for real-time large-scale 3D point cloud processing. Future research may extend this iterative dual-representation framework to other 3D tasks such as object detection and motion prediction.

Summary updated

Abstract

We present a novel and flexible architecture for point cloud segmentation with dual-representation iterative learning. In point cloud processing, different representations have their own pros and cons. Thus, finding suitable ways to represent point cloud data structure while keeping its own internal physical property such as permutation and scale-invariant is a fundamental problem. Therefore, we propose our work, DRINet, which serves as the basic network structure for dual-representation learning with great flexibility at feature transferring and less computation cost, especially for large-scale point clouds. DRINet mainly consists of two modules called Sparse Point-Voxel Feature Extraction and Sparse Voxel-Point Feature Extraction. By utilizing these two modules iteratively, features can be propagated between two different representations. We further propose a novel multi-scale pooling layer for pointwise locality learning to improve context information propagation. Our network achieves state-of-the-art results for point cloud classification and segmentation tasks on several datasets while maintaining high runtime efficiency. For large-scale outdoor scenarios, our method outperforms state-of-the-art methods with a real-time inference time of 62ms per frame.
BibTeX
@inproceedings{ye2021drinet,
  author = {Maosheng Ye and Shuangjie Xu and Tongyi Cao and Qifeng Chen},
  title = {{DRINet: A Dual-Representation Iterative Learning Network for Point Cloud Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=syhPTKjbMhz},
}
Figure