Robust Fine-tuning for Pre-trained 3D Point Cloud Models

Zhibo Zhang, Ximing Yang, Weizhong Zhang, Cheng JinarXiv 2024 · DOI · arXiv

Abstract

Fine-tuning pre-trained 3D point cloud models often results in a "robustness-accuracy trade-off," where gains in task-specific performance come at the cost of generalizability. We present WiSE-FT-LP, a three-stage method combining weight-space interpolation with linear probing. By first performing full fine-tuning, then interpolating weights between pre-trained and fine-tuned backbones, and finally re-training the classification head via linear probing, we achieve models that are both highly accurate on target datasets and resilient to distribution shifts. Our results on ScanObjectNN and ModelNet40 demonstrate that WiSE-FT-LP consistently outperforms standard fine-tuning and existing ensemble methods.

Introduction

In 3D computer vision, the standard paradigm involves pre-training models on massive datasets like ShapeNet to learn robust, general-purpose features. However, adapting these models to specific downstream tasks, such as classifying real-world objects in ScanObjectNN, often leads to feature distortion. While the model's accuracy on the target task improves, its general robustness—its ability to handle noise and environmental variations—typically plummets. This paper addresses this challenge by proposing a method to reconcile task-specific performance with pre-trained robustness.

Methodology: WiSE-FT-LP

The authors introduce Weight-Space Ensembles for Fine-Tuning then Linear Probing (WiSE-FT-LP). This approach consists of three distinct stages designed to preserve pre-trained knowledge while adapting to new data:

  1. Full Fine-Tuning: A task-specific head is added to a pre-trained backbone ($\theta_{PT}$), and the entire model is fine-tuned on the target dataset to obtain $\theta_{FT}$.
  2. Weight-Space Interpolation: An "ensembled" backbone ($\theta_{WiSE}$) is created by linearly interpolating the weights of the original pre-trained backbone and the fine-tuned backbone using a mixing coefficient $\alpha$.
  3. Linear Probing: The ensembled backbone is frozen, and only the classification head is re-trained. This step is crucial for calibrating the head to the hybrid feature space created by the interpolation.
Overview of the WiSE-FT-LP three-step process.

Experimental Setup

The method was evaluated using mainstream architectures like Point-M2AE and ReCon across several key datasets:

  • ShapeNet: Used for initial large-scale pre-training.
  • ScanObjectNN: The primary target for downstream fine-tuning, featuring real-world challenges like background noise and geometric transformations (PB_T50_RS, OBJ_ONLY, OBJ_BG).
  • ModelNet40: Utilized to measure "Backbone Robustness" by evaluating the representational power of the backbone features using a linear SVM.

Results and Analysis

A key discovery in this work is the "Nadir" of interpolation: simply blending weights can sometimes lead to a significant drop in accuracy at certain mixing coefficients, indicating a conflict between pre-training and downstream optimization directions. WiSE-FT-LP mitigates this by using the final linear probing step to "reconcile" these features.

Empirical results show a superior balance between accuracy and robustness. For instance, using Point-M2AE on the PB_T50_RS task, WiSE-FT-LP achieved 86.99% accuracy and a 90.68% robustness score, outperforming standard full fine-tuning (86.43% accuracy, 89.59% robustness).

Accuracy vs. Robustness trade-off for WiSE-FT and WiSE-FT-LP.

As shown in the figure above, the addition of the Linear Probing step (WiSE-FT-LP) consistently yields a better trade-off curve compared to the standard WiSE-FT method, particularly in maintaining higher overall accuracy as robustness increases.

Conclusion

WiSE-FT-LP provides a robust framework for fine-tuning 3D point cloud models without adding inference-time overhead. By anchoring the model to its pre-trained roots while carefully calibrating the task head, it achieves state-of-the-art performance and resilience. Future work will focus on automating the selection of the mixing coefficient $\alpha$ and exploring the method's applicability to more complex tasks like segmentation and detection.

Summary updated

BibTeX
@misc{zhang2024robust,
  author = {Zhibo Zhang and Ximing Yang and Weizhong Zhang and Cheng Jin},
  title = {{Robust Fine-tuning for Pre-trained 3D Point Cloud Models}},
  year = {2024},
  eprint = {2404.16422},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.16422},
}
Figure