Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation

Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia

Abstract

The Hierarchical Point-Edge Interaction Network (HPEIN) addresses the challenges of 3D point cloud semantic segmentation by explicitly modeling the relationships between neighboring points. Unlike traditional methods that often discard relational context through simple aggregation, HPEIN utilizes a dual-branch architecture that facilitates bidirectional message passing between point and edge features across multiple scales. By incorporating an auxiliary edge loss to enforce semantic consistency, the model achieves superior performance on major benchmarks, particularly in defining object boundaries.

The Challenge of Point Cloud Segmentation

Semantic segmentation in 3D—assigning category labels like "chair" or "wall" to every point—is inherently difficult due to the irregular, sparse, and unordered nature of point clouds. While early architectures like PointNet++ introduced hierarchical grouping, they often rely on max-pooling to aggregate local information, which can lose the nuanced relational data between points. To truly understand a scene, a network must model whether connected points belong to the same object, a concept the authors term "semantic consistency."

Methodology: Hierarchical Point-Edge Interaction

The researchers propose a two-branch framework that interacts at every level of the network hierarchy.

Dual-Branch Architecture

The core of HPEIN is the interaction between two distinct branches:

  1. The Point Branch: An encoder-decoder structure that extracts and upsamples point-wise features.
  2. The Edge Branch: A novel hierarchical branch that constructs a graph of the scene, starting at a coarse resolution and upsampling edges as point resolution increases.
Conceptual overview of the dual-branch interaction between Point and Edge branches across layers.

Hierarchical Interaction and Edge Upsampling

Instead of reconstructing a graph from scratch at every layer, HPEIN uses an Edge Upsampling scheme. Edge features from coarser layers are propagated to finer ones using spatial distance-based interpolation. This allows for long-range message passing without the computational burden of full graph construction at high resolutions.

Detailed architecture of the HPEIN decoding stage, showing the hierarchical interaction between Point Modules and Edge Modules.

At each layer, point features are fed into the edge branch to generate edge features, while edge features are max-pooled and returned to the point branch to provide "regional guidance" for point-wise predictions.

Auxiliary Edge Loss

To ensure edge features capture meaningful semantic boundaries, the model is trained with an auxiliary task: predicting whether an edge connects points of the same category (intra-class) or different categories (inter-class). This forces the network to learn more discriminative features specifically at the borders between objects.

Key Insights

  • Edges as Descriptors: Edges are treated as distinct features capturing local geometric and semantic "texture," rather than just paths for convolution.
  • Efficiency through Hierarchy: Hierarchical graph construction and edge upsampling enable the model to handle large point clouds with manageable memory overhead.
  • Boundary Regularization: The auxiliary edge loss acts as a powerful regularizer, sharpening the network's ability to distinguish between adjacent objects.

Evaluation and Results

HPEIN was evaluated on the S3DIS and ScanNet v2 datasets, setting new benchmarks at the time of publication:

  • S3DIS (Area 5): Achieved an mIoU of 61.85%, significantly outperforming previous state-of-the-art models like PCCN and PointCNN.
  • ScanNet v2: Reached an mIoU of 61.8%, a substantial improvement over PointNet++ (33.9%) and PointConv (55.6%).

Qualitative analysis confirms that the model is particularly effective at "cleaning up" semantic boundaries and identifying small, detailed structures that are often blurred by other methods.

Conclusion

By elevating edges to "first-class citizens" and modeling their interaction with point features hierarchically, HPEIN provides a robust solution for 3D semantic segmentation. While the current localized graph construction offers efficiency, future work may explore multi-range edge connections to capture even broader global contexts.

Summary updated

Abstract

We achieve 3D semantic scene labeling by exploring semantic relation between each point and its contextual neighbors through edges. Besides an encoder-decoder branch for predicting point labels, we construct an edge branch to hierarchically integrate point features and generate edge features. To incorporate point features in the edge branch, we establish a hierarchical graph framework, where the graph is initialized from a coarse layer and gradually enriched along the point decoding process. For each edge in the final graph, we predict a label to indicate the semantic consistency of the two connected points to enhance point prediction. At different layers, edge features are also fed into the corresponding point module to integrate contextual information for message passing enhancement in local regions. The two branches interact with each other and cooperate in segmentation. Decent experimental results on several 3D semantic labeling datasets demonstrate the effectiveness of our work.
BibTeX
@inproceedings{jiang2019hierarchical,
  author = {Li Jiang and Hengshuang Zhao and Shu Liu and Xiaoyong Shen and Chi-Wing Fu and Jiaya Jia},
  title = {{Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.01053},
}
Figure