Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning

Yang Liu, Chen Chen, Can Wang, Xulin King, Mengyuan Liu

Abstract

Point-RAE (Point Regress AutoEncoder) introduces a "mask regressor" to decouple representation learning from reconstruction in 3D point cloud models. By predicting masked features before reconstructing them and utilizing this regressor during fine-tuning, the model achieves state-of-the-art performance on challenging benchmarks like ScanObjectNN (90.28% accuracy).

Introduction

Masked Autoencoders (MAE) have become a standard for self-supervised learning in computer vision. The core idea involves masking parts of the input data and training a model to reconstruct the missing components. However, current 3D MAE architectures suffer from two primary limitations: incomplete functional decoupling, where the decoder's reconstruction task "pollutes" the encoder's latent space, and wasted knowledge, as the decoder is typically discarded after pre-training.

Methodology: The RAE Paradigm

To address these issues, the authors propose Point-RAE, a new paradigm that introduces an intermediate "mask regressor" step to separate high-level feature prediction from low-level coordinate reconstruction.

Comparison of the existing MAE paradigm and the proposed RAE paradigm.

Mask Regressor and Alignment

Instead of feeding visible patches and learnable "mask tokens" directly into a decoder, Point-RAE utilizes a dedicated Mask Regressor. This component predicts the latent representations of masked patches based on the encoder's output. To ensure semantic consistency, an Alignment Constraint is applied, comparing the regressor's predictions against features generated by an Exponential Moving Average (EMA) version of the encoder. This forces the regressor to operate within the same semantic space as the encoder.

Decoupled Reconstruction and Fine-tuning

The reconstruction task is decoupled: a lightweight decoder takes only the predicted masked representations to reconstruct original point coordinates. Crucially, unlike standard MAEs, Point-RAE retains the mask regressor during fine-tuning. By concatenating features from both the encoder and the regressor, the model can effectively "hallucinate" missing information, serving as a "densitifier" for sparse 3D data.

Key Results

Point-RAE sets new state-of-the-art (SOTA) records across multiple benchmarks. On the challenging ScanObjectNN (PB-T50-RS) dataset, it achieved 90.28% accuracy, surpassing previous methods like I2P-MAE and Point-M2AE. It also reached 94.1% accuracy on ModelNet40 and showed exceptional generalization in few-shot learning scenarios.

Training curves showing the impact of the regressor on pre-training loss and fine-tuning accuracy.

As shown in the training curves, the inclusion of the regressor significantly improves both the pre-training reconstruction loss and the final downstream task accuracy. The model with the regressor (red lines) consistently outperforms the baseline (blue lines), validating the "regress before construct" approach.

Conclusion

Point-RAE demonstrates that predicting features before rendering points leads to more robust global structures in 3D point cloud models. While the computational overhead of the regressor during fine-tuning warrants further study, the performance gains across classification and segmentation tasks establish it as a powerful new paradigm for self-supervised 3D learning.

Summary updated

Abstract

Masked Autoencoders (MAE) have demonstrated promising performance in self-supervised learning for both 2D and 3D computer vision. Nevertheless, existing MAE-based methods still have certain drawbacks. Firstly, the functional decoupling between the encoder and decoder is incomplete, which limits the encoder's representation learning ability. Secondly, downstream tasks solely utilize the encoder, failing to fully leverage the knowledge acquired through the encoder-decoder architecture in the pre-text task. In this paper, we propose Point Regress AutoEncoder (Point-RAE), a new scheme for regressive autoencoders for point cloud self-supervised learning. The proposed method decouples functions between the decoder and the encoder by introducing a mask regressor, which predicts the masked patch representation from the visible patch representation encoded by the encoder and the decoder reconstructs the target from the predicted masked patch representation. By doing so, we minimize the impact of decoder updates on the representation space of the encoder. Moreover, we introduce an alignment constraint to ensure that the representations for masked patches, predicted from the encoded representations of visible patches, are aligned with the masked patch presentations computed from the encoder. To make full use of the knowledge learned in the pre-training stage, we design a new finetune mode for the proposed Point-RAE. Extensive experiments demonstrate that our approach is efficient during pre-training and generalizes well on various downstream tasks. Specifically, our pre-trained models achieve a high accuracy of \textbf{90.28\%} on the ScanObjectNN hardest split and \textbf{94.1\%} accuracy on ModelNet40, surpassing all the other self-supervised learning methods. Our code and pretrained model are public available at: \url{https://github.com/liuyyy111/Point-RAE}.
BibTeX
@inproceedings{liu2023regress,
  author = {Yang Liu and Chen Chen and Can Wang and Xulin King and Mengyuan Liu},
  title = {{Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612106},
}
Figure