Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching

Yikai Bian, Le Hui, Jianjun Qian, Jin Xie

TL;DR

This paper introduces a novel framework for Unsupervised Domain Adaptation (UDA) in 3D point cloud segmentation by shifting the focus from global distribution alignment to local-level graph matching. By dynamically constructing local feature graphs and using Optimal Transport to match them against a source-domain memory bank, the model learns to bridge the "domain gap" (e.g., differences in LiDAR beam density) while preserving the distinct semantic boundaries of different object categories.

What Problem Does This Paper Address?

Deep learning models for 3D point cloud segmentation are notoriously sensitive to the specific sensor used to collect data. A model trained on high-density, 64-beam LiDAR data (the source domain) often fails when deployed on lower-density, 32-beam data (the target domain) due to domain shift.

Existing UDA methods typically try to align the global feature distributions of the two domains. However, the authors argue that global alignment is too coarse; it often leads to "semantic ambiguity," where features from different categories (like a pole vs. a tree) get blurred together in the latent space. The challenge is to align the domains at a local, geometric level so that the model recognizes that a "car-shaped" cluster of points in the source domain corresponds to a "car-shaped" cluster in the target domain, regardless of point density.

Methods

The authors propose a three-pronged approach to local alignment:

  1. Dynamic Feature Graphs: Instead of looking at the whole scene, the model uses Farthest Point Sampling (FPS) to select centroid points and builds local graphs around them. These graphs capture the k-nearest neighbors and their relative distances, effectively encoding local geometric structures.
  2. Feature Graph Memory Bank: To overcome the limited variety of objects present in a single training batch, the authors maintain a memory bank of source-domain feature graphs. This ensures the model always has a rich "dictionary" of labeled source features to compare against.
  3. Graph Matching via Optimal Transport: To find which source graph matches a given target graph, the authors use the Sinkhorn algorithm to solve an Optimal Transport problem. This produces an assignment matrix that provides point-level correspondences between the two domains.
  4. Category-Guided Contrastive Loss: To further sharpen the model, they implement a contrastive loss. It pushes target features closer to "positive" (same-category) source features and further away from "negative" (different-category) ones, increasing the discriminative power of the model on unlabeled data.

The system uses PointWeb as its backbone architecture and can be further boosted using a two-stage self-training strategy with pseudo-labels.

Datasets

The researchers evaluated their method across three distinct domain adaptation scenarios:

  • vKITTI $\to$ SemanticPOSS: Adapting from synthetic (virtual) outdoor scenes to real-world LiDAR data.
  • S3DIS $\to$ ScanNet: An indoor-to-indoor adaptation between two popular room-scale datasets.
  • SemanticKITTI $\to$ nuScenes: A real-to-real outdoor adaptation involving different LiDAR hardware (64-beam vs. 32-beam).

Key Insights and Ideas

  • Local > Global: The core "aha moment" is that local geometric consistency is more reliable than global statistical consistency when sensors change. By matching local neighborhoods, the model retains the "shape" of the features.
  • Memory as a Stabilizer: The memory bank prevents the model from "forgetting" rare classes during training and provides a stable reference for the Sinkhorn matching process.
  • Principled Matching: Using Optimal Transport (Sinkhorn) is a mathematically rigorous way to handle the "unstructured" nature of point clouds, allowing the model to find the best possible alignment between two sets of points that don't have a fixed order.

Key Results

The proposed method achieved state-of-the-art (SOTA) performance across all tested benchmarks, measured by mean Intersection-over-Union (mIoU):

  • vKITTI $\to$ SemanticPOSS: Reached 54.9% mIoU, significantly outperforming the "Source Only" baseline (44.6%) and other UDA methods like ADDA (49.8%).
  • S3DIS $\to$ ScanNet: Achieved 53.8% mIoU, a massive jump from the baseline (43.2%).
  • SemanticKITTI $\to$ nuScenes: Reached 37.3% mIoU, beating the previous best-reported method (C&L) which sat at 31.6%.

Qualitative visualizations (t-SNE plots) confirmed that the method successfully clusters target-domain features by category much more cleanly than baseline approaches.

Limitations and Future Work

  • Computational Overhead: While the Sinkhorn algorithm is efficient, performing graph matching against a memory bank for every centroid point adds computational complexity during the training phase.
  • Backbone Dependency: The method was primarily tested with PointWeb. While the logic is general, its performance might vary with different backbone architectures (like Sparse Convolution or Transformers).
  • Future Directions: The authors suggest that this graph-matching approach could be extended to multi-modal scenarios (e.g., fusing 2D images with 3D point clouds) to further harden the model against domain shift.

Summary updated

Abstract

Unsupervised domain adaptation for point cloud semantic segmentation has attracted great attention due to its effectiveness in learning with unlabeled data. Most of existing methods use global-level feature alignment to transfer the knowledge from the source domain to the target domain, which may cause the semantic ambiguity of the feature space. In this paper, we propose a graph-based framework to explore the local-level feature alignment between the two domains, which can reserve semantic discrimination during adaptation. Specifically, in order to extract local-level features, we first dynamically construct local feature graphs on both domains and build a memory bank with the graphs from the source domain. In particular, we use optimal transport to generate the graph matching pairs. Then, based on the assignment matrix, we can align the feature distributions between the two domains with the graph-based local feature loss. Furthermore, we consider the correlation between the features of different categories and formulate a category-guided contrastive loss to guide the segmentation model to learn discriminative features on the target domain. Extensive experiments on different synthetic-to-real and real-to-real domain adaptation scenarios demonstrate that our method can achieve state-of-the-art performance.
BibTeX
@inproceedings{bian2022unsupervised,
  author = {Yikai Bian and Le Hui and Jianjun Qian and Jin Xie},
  title = {{Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981603},
}

Loading related content...

Figure