Towards Semi-supervised Dual-modal Semantic Segmentation

Qiulei Dong, Jianan Li, Shuang Deng

Abstract

With the development of 3D and 2D data acquisition techniques, it has become easy to obtain point clouds and images of scenes simultaneously, which further facilitates dual-modal semantic segmentation. Most existing methods for simultaneously segmenting point clouds and images rely heavily on the quantity and quality of the labeled training data. However, massive point-wise and pixel-wise labeling procedures are time-consuming and labor-intensive. To address this issue, we propose a parallel dual-stream network to handle the semi-supervised dual-modal semantic segmentation task, called PD-Net, by jointly utilizing a small number of labeled point clouds, a large number of unlabeled point clouds, and unlabeled images. The proposed PD-Net consists of two parallel streams (called original stream and pseudo-label prediction stream). The pseudo-label prediction stream predicts the pseudo labels of unlabeled point clouds and their corresponding images. Then, the unlabeled data is sent to the original stream for self-training. Each stream contains two encoder-decoder branches for 3D and 2D data respectively. In each stream, multiple dual-modal fusion modules are explored for fusing the dual-modal features. In addition, a pseudo-label optimization module is explored to optimize the pseudo labels output by the pseudo-label prediction stream. Experimental results on two public datasets demonstrate that the proposed PD-Net not only outperforms the comparative semi-supervised methods but also achieves competitive performances with some fully-supervised methods in most cases.

Introduction

Semantic segmentation—the task of assigning a category to every pixel in an image or point in a 3D space—is a cornerstone of autonomous driving and robotics. While integrating 2D images (rich in texture) and 3D point clouds (rich in geometry) provides superior results, it introduces a significant data bottleneck. Labeling 3D scenes point-by-point is exceptionally labor-intensive and expensive.

Most current dual-modal research assumes the availability of fully labeled datasets. Conversely, existing semi-supervised methods typically focus on a single modality. PD-Net addresses this gap by asking: How can we leverage the complementary strengths of 2D and 3D data to train a model when the vast majority of the data is unlabeled?

Methodology: The PD-Net Architecture

The proposed PD-Net (Parallel Dual-stream Network) is designed to exploit the synergy between modalities through three primary components:

  1. Parallel Dual-Streams: The network utilizes an "Original Stream" and a "Pseudo-label Prediction Stream." The latter acts as a stable "teacher" whose weights are updated as an Exponential Moving Average (EMA) of the Original Stream. This teacher generates targets for the student (Original Stream) to learn from using unlabeled data.
  2. Dual-modal Fusion (DMF) Module: Rather than simple concatenation, PD-Net employs a multi-head attention mechanism. This allows the model to adaptively weight modalities based on the region—for instance, prioritizing 2D data for color-rich objects and 3D data for structural elements like walls.
  3. Pseudo-label Optimization (PLO) Module: To mitigate the noise inherent in self-generated labels, the PLO module uses a cross-modal voting mechanism. A 3D pseudo-label is only retained if it aligns with the projected 2D prediction or if the model exhibits high confidence. Disagreements between modalities lead to label discarding, preventing the propagation of incorrect information.
The architecture of the proposed PD-Net (a) and the detailed structure of its internal streams (b).

Experimental Evaluation

The model was evaluated on the ScanNet indoor dataset (1,613 scenes) and the NYUv2 RGB-D dataset. A key insight from the experiments is the effectiveness of Cross-Modal Validation. Because 2D and 3D sensors fail in different ways (e.g., images struggle with occlusion while point clouds struggle with fine texture), requiring consistency between them filters out common semi-supervised "hallucinations."

Key Results

  • Efficiency: With only 20% of labels, PD-Net achieved a 3D mIoU of 63.38% on ScanNet, performing competitively with state-of-the-art fully supervised models.
  • Ablation Studies: Removing the PLO or DMF modules resulted in significant performance degradation, confirming that cross-modal voting and attention-based fusion are critical to the system's success.
Qualitative comparison of semantic segmentation results on the ScanNet dataset, showing PD-Net's performance with 10% and 20% labeled data compared to ground truth and baselines.

Conclusion

PD-Net demonstrates that the marginal utility of human labeling decreases rapidly when cross-modal consistency is effectively leveraged. By using 2D and 3D data as "checks and balances" for each other, the framework achieves high accuracy with minimal supervision. Future work will focus on reducing the computational cost of the dual-stream architecture and scaling the approach to sparser outdoor LiDAR datasets.

Summary updated

Abstract

With the development of 3D and 2D data acquisition techniques, it has become easy to obtain point clouds and images of scenes simultaneously, which further facilitates dual-modal semantic segmentation. Most existing methods for simultaneously segmenting point clouds and images rely heavily on the quantity and quality of the labeled training data. However, massive point-wise and pixel-wise labeling procedures are time-consuming and labor-intensive. To address this issue, we propose a parallel dual-stream network to handle the semi-supervised dual-modal semantic segmentation task, called PD-Net, by jointly utilizing a small number of labeled point clouds, a large number of unlabeled point clouds, and unlabeled images. The proposed PD-Net consists of two parallel streams (called original stream and pseudo-label prediction stream). The pseudo-label prediction stream predicts the pseudo labels of unlabeled point clouds and their corresponding images. Then, the unlabeled data is sent to the original stream for self-training. Each stream contains two encoder-decoder branches for 3D and 2D data respectively. In each stream, multiple dual-modal fusion modules are explored for fusing the dual-modal features. In addition, a pseudo-label optimization module is explored to optimize the pseudo labels output by the pseudo-label prediction stream. Experimental results on two public datasets demonstrate that the proposed PD-Net not only outperforms the comparative semi-supervised methods but also achieves competitive performances with some fully-supervised methods in most cases.
BibTeX
@article{dong2025towards,
  author = {Qiulei Dong and Jianan Li and Shuang Deng},
  title = {{Towards Semi-supervised Dual-modal Semantic Segmentation}},
  journal = {IEEE Transactions on Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3604939},
}
Figure