UniDxMD: Towards Unified Representation for Cross-Modal Unsupervised Domain Adaptation in 3D Semantic Segmentation

Zhengyin Liang, Hui Yin, Min Liang, Qianqian Du, Ying Yang, Hua HuangICCV 2025

TL;DR

This paper introduces UniDxMD, a novel method for 3D semantic segmentation that tackles the challenge of domain shifts by learning a unified discrete representation across different sensor modalities (like LiDAR and cameras) and data distributions. By quantizing heterogeneous data into a shared latent space, UniDxMD significantly improves robustness and achieves state-of-the-art performance in cross-modal unsupervised domain adaptation tasks.

What Problem Does This Paper Address?

3D semantic segmentation, crucial for applications like autonomous driving and robotics, often suffers from performance degradation when the data distribution shifts between training and deployment (e.g., different sensors, lighting conditions, or environments). This is particularly challenging in cross-modal scenarios where data comes from multiple sources (e.g., LiDAR and cameras) that may have different characteristics. Existing methods often address cross-modal and cross-domain adaptation separately, leading to suboptimal results and overlooking the inherent semantic connections between different data types. UniDxMD aims to bridge this gap by unifying these adaptation objectives into a single framework that learns a robust representation invariant to modality and domain variations.

Methods

The core of UniDxMD lies in learning a unified discrete representation. This is achieved through two main novel components:

  • Cluster-Based Soft Quantization Mechanism (CSQM): Instead of traditional hard vector quantization (which uses a single nearest codebook entry and is non-differentiable), CSQM uses a differentiable approach. It quantizes encoded features into weighted combinations of local code vectors. This is done by calculating an affinity matrix based on a Gaussian kernel and then using a mask to select relevant code vectors. This soft quantization allows for gradient flow, enabling end-to-end training and preventing codebook collapse.
  • Latent Space Regularization (LSR): To structure the learned latent space and make it semantically meaningful, LSR uses "joint prototypes" (derived from pseudo-labels on target data) to guide the update of code vectors. This process encourages code vectors with similar semantics to aggregate, leading to a more compact and discriminative latent space. It also incorporates a joint relation constraint loss to maintain semantic consistency across modalities and domains.

Datasets

The paper evaluates UniDxMD on four challenging cross-modal unsupervised domain adaptation (UDA) scenarios using public datasets:

  • nuScenes: Used for scene layout variations (USA/Sing.) and lighting changes (Day/Night).
  • Virtual KITTI (v.KITTI) / Semantic KITTI (Sem.KITTI): Used for synthetic-to-real domain adaptation.
  • A2D2 / Semantic KITTI (Sem.KITTI): Used for sensor setup variations.

These datasets provide synchronized LiDAR and camera data, allowing for 3D-to-2D projections and evaluation of cross-modal adaptation.

Key Insights and Ideas

  • Unified Representation is Key: The paper argues that both modality and domain adaptation fundamentally aim to extract semantically equivalent information. Therefore, learning a single, unified discrete representation is a more effective strategy than treating these as separate objectives.
  • Soft Quantization for Differentiability: Moving away from hard VQ is crucial for enabling end-to-end training and avoiding issues like codebook collapse, which plague traditional VQ methods in complex adaptation tasks.
  • Structured Latent Space via Prototypes: Using semantic prototypes (especially joint ones derived from target domain predictions) provides a powerful mechanism to guide the learning of the discrete latent space, making it more organized and semantically rich.
  • Robustness to Domain Shifts: By learning representations that are invariant to modality and domain variations, UniDxMD demonstrates significantly improved performance on challenging UDA tasks.

Key Results

UniDxMD achieves state-of-the-art performance across all four evaluated UDA scenarios.

  • Quantitative: It outperforms the previous state-of-the-art method, xMUDA, by significant margins (e.g., 4.9% to 10.5% in ensemble xM scores). In the nuScenes: Day/Night scenario, it surpasses the second-best method by 2.5%.
  • Qualitative: Visualizations show that UniDxMD learns a more structured and utilized latent space compared to existing methods, leading to better segmentation results, especially in challenging conditions.
  • Scalability: Integrating UniDxMD with visual foundation models (like CLIP) shows competitive results, highlighting its potential for future advancements.

Limitations and Future Work

  • Codebook Size Sensitivity: While the method is not highly sensitive, the paper notes that codebook size and feature dimensions have an impact, with optimal ranges identified.
  • Pseudo-labeling Noise: The method relies on pseudo-labels for target domain learning, which can introduce noise. The paper mentions not applying further post-processing to pseudo-labels, suggesting this could be an area for improvement.
  • Future Work: The authors suggest exploring the potential of leveraging visual foundation models (VFMs) more deeply to guide latent space learning, indicating a promising direction for future research.

Summary updated

BibTeX
@inproceedings{liang2025unidxmd,
  author = {Zhengyin Liang and Hui Yin and Min Liang and Qianqian Du and Ying Yang and Hua Huang},
  title = {{UniDxMD: Towards Unified Representation for Cross-Modal Unsupervised Domain Adaptation in 3D Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure