Locating and Mitigating Gradient Conflicts in Point Cloud Domain Adaptation via Saliency Map Skewness

Jiaqi Tang, Yinsong Xu, Qingchao ChenarXiv 2025 · DOI · arXiv

Introduction

In 3D computer vision, models often face a "domain gap" when transitioning from synthetic training data (e.g., CAD models) to real-world lidar scans. Unsupervised Domain Adaptation (UDA) aims to bridge this gap without target-domain labels. A common strategy involves Multi-Task Learning (MTL), where a model performs a primary classification task alongside self-supervised auxiliary tasks. However, these tasks can produce conflicting gradients, leading to "negative transfer" where auxiliary learning actually degrades performance on specific classes. This paper introduces a novel method to identify and mitigate these conflicts by analyzing the statistical properties of model attention.

Methodology: The SM-DSB Framework

The researchers propose the Saliency Map-based Data Sampling Block (SM-DSB), a plug-and-play module designed to filter out detrimental training samples. The framework operates through two primary components: the Measurer and the Selector.

The SM-DSB architecture, highlighting the Measurer (calculating skewness) and the Selector (filtering SSL gradients) within the UDA pipeline.
  1. The Measurer: For each input point cloud, the Measurer generates a saliency map—a heatmap indicating the importance of specific points to the model's prediction. It then calculates the skewness (a 3rd-order statistic) of this map. High skewness indicates the model is hyper-focused on a few points (shortcut learning), while low skewness suggests a more robust, distributed focus.
  2. The Selector: Using an adaptive threshold based on the calculated skewness, the Selector identifies samples likely to cause gradient conflicts. It then "filters" these samples, preventing their self-supervised gradients from interfering with the model update during that training batch.

The Skewness-Conflict Correlation

A central contribution of this work is the discovery of a strong correlation between saliency map skewness and gradient conflict. When a model's attention is highly skewed, it becomes vulnerable to domain shifts—for instance, if a real-world sensor misses the specific points the model has over-indexed on.

Conceptual overview of the relationship between saliency map skewness and gradient conflict, alongside a correlation plot showing the negative relationship between skewness and cosine similarity gain.

As shown in the correlation analysis, there is a clear negative relationship (Correlation Coefficient = -0.63) between skewness and the cosine similarity gain of gradients relative to an "oracle" gradient (the ideal gradient path if labels were available). By targeting high-skewness samples, the SM-DSB effectively performs "gradient surgery" to keep the training trajectory aligned with the optimal path.

Experimental Results

The SM-DSB was evaluated on two major benchmarks: PointDA-10 (classification) and PointSegDA (segmentation).

  • State-of-the-Art Performance: Integrated into the "Self-dist-GCN" framework, SM-DSB achieved top results in 5 out of 6 PointDA-10 tasks. Notably, it reached 90.5% accuracy on the challenging Real-to-Synthetic (ScanNet to ModelNet) task.
  • Class-Level Gains: The method significantly improved accuracy for difficult classes. For example, accuracy for the "plant" class increased by nearly 20% compared to baselines that did not account for gradient interference.
  • Segmentation Robustness: On the PointSegDA benchmark, the method improved the mean Intersection over Union (mIoU) in 10 out of 12 transfer directions, demonstrating its effectiveness across different 3D tasks.

Conclusion

This research demonstrates that "more data" is not always beneficial in multi-task UDA; rather, the quality and alignment of gradients are paramount. By using saliency map skewness as a proxy for gradient conflict, the SM-DSB provides an effective, label-free way to mitigate negative transfer. Future work will explore additional statistical metrics and the application of this technique to 2D image domain adaptation.

Summary updated

Abstract

Object classification models utilizing point cloud data are fundamental for 3D media understanding, yet they often struggle with unseen or out-of-distribution (OOD) scenarios. Existing point cloud unsupervised domain adaptation (UDA) methods typically employ a multi-task learning (MTL) framework that combines primary classification tasks with auxiliary self-supervision tasks to bridge the gap between cross-domain feature distributions. However, our further experiments demonstrate that not all gradients from self-supervision tasks are beneficial and some may negatively impact the classification performance. In this paper, we propose a novel solution, termed Saliency Map-based Data Sampling Block (SM-DSB), to mitigate these gradient conflicts. Specifically, our method designs a new scoring mechanism based on the skewness of 3D saliency maps to estimate gradient conflicts without requiring target labels. Leveraging this, we develop a sample selection strategy that dynamically filters out samples whose self-supervision gradients are not beneficial for the classification. Our approach is scalable, introducing modest computational overhead, and can be integrated into all the point cloud UDA MTL frameworks. Extensive evaluations demonstrate that our method outperforms state-of-the-art approaches. In addition, we provide a new perspective on understanding the UDA problem through back-propagation analysis.
BibTeX
@misc{tang2025locating,
  author = {Jiaqi Tang and Yinsong Xu and Qingchao Chen},
  title = {{Locating and Mitigating Gradient Conflicts in Point Cloud Domain Adaptation via Saliency Map Skewness}},
  year = {2025},
  eprint = {2504.15796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.15796},
}
Figure