Abstract
Point-RAE (Point Regress AutoEncoder) introduces a "mask regressor" to decouple representation learning from reconstruction in 3D point cloud models. By predicting masked features before reconstructing them and utilizing this regressor during fine-tuning, the model achieves state-of-the-art performance on challenging benchmarks like ScanObjectNN (90.28% accuracy).
Introduction
Masked Autoencoders (MAE) have become a standard for self-supervised learning in computer vision. The core idea involves masking parts of the input data and training a model to reconstruct the missing components. However, current 3D MAE architectures suffer from two primary limitations: incomplete functional decoupling, where the decoder's reconstruction task "pollutes" the encoder's latent space, and wasted knowledge, as the decoder is typically discarded after pre-training.
Methodology: The RAE Paradigm
To address these issues, the authors propose Point-RAE, a new paradigm that introduces an intermediate "mask regressor" step to separate high-level feature prediction from low-level coordinate reconstruction.
Mask Regressor and Alignment
Instead of feeding visible patches and learnable "mask tokens" directly into a decoder, Point-RAE utilizes a dedicated Mask Regressor. This component predicts the latent representations of masked patches based on the encoder's output. To ensure semantic consistency, an Alignment Constraint is applied, comparing the regressor's predictions against features generated by an Exponential Moving Average (EMA) version of the encoder. This forces the regressor to operate within the same semantic space as the encoder.
Decoupled Reconstruction and Fine-tuning
The reconstruction task is decoupled: a lightweight decoder takes only the predicted masked representations to reconstruct original point coordinates. Crucially, unlike standard MAEs, Point-RAE retains the mask regressor during fine-tuning. By concatenating features from both the encoder and the regressor, the model can effectively "hallucinate" missing information, serving as a "densitifier" for sparse 3D data.
Key Results
Point-RAE sets new state-of-the-art (SOTA) records across multiple benchmarks. On the challenging ScanObjectNN (PB-T50-RS) dataset, it achieved 90.28% accuracy, surpassing previous methods like I2P-MAE and Point-M2AE. It also reached 94.1% accuracy on ModelNet40 and showed exceptional generalization in few-shot learning scenarios.
As shown in the training curves, the inclusion of the regressor significantly improves both the pre-training reconstruction loss and the final downstream task accuracy. The model with the regressor (red lines) consistently outperforms the baseline (blue lines), validating the "regress before construct" approach.
Conclusion
Point-RAE demonstrates that predicting features before rendering points leads to more robust global structures in 3D point cloud models. While the computational overhead of the regressor during fine-tuning warrants further study, the performance gains across classification and segmentation tasks establish it as a powerful new paradigm for self-supervised 3D learning.
Summary updated