MATE: Masked Autoencoders are Online 3D Test-Time Learners

M. Jehanzeb Mirza, Inkyu Shin, Wei Lin, Andreas Schriebl, Kunyang Sun, Jaesung Choe, Horst Possegger, Mateusz Kozinski, In So Kweon, Kun-Jin Yoon, Horst Bischof

TL;DR

MATE is the first test-time training (TTT) method specifically designed to make 3D point cloud classifiers robust against real-world distribution shifts online. By leveraging a Masked Autoencoder (MAE) objective at test time, MATE masks up to 95% of incoming point cloud data to efficiently adapt the model on-the-fly, achieving state-of-the-art robustness while remaining lightweight enough for real-time applications.

What Problem Does This Paper Address?

Deep learning models excel at classifying 3D point clouds under ideal conditions, but their performance plummets when faced with real-world distribution shifts. In practical scenarios—like autonomous driving or robotic navigation—LiDAR data is frequently corrupted by sensor noise, environmental factors, missing data, or geometric distortions.

While Test-Time Training (TTT) has emerged as a successful paradigm in 2D computer vision to adapt models to out-of-distribution (OOD) data on-the-fly, simply porting 2D TTT methods (such as entropy minimization or rotation prediction) to 3D data fails. 3D point clouds require a deep understanding of spatial geometry that 2D methods do not capture. This paper introduces a 3D-specific TTT framework capable of unsupervised, real-time adaptation to diverse and unpredictable corruptions.

Methods

MATE builds upon the PointMAE (Point Masked Autoencoder) architecture, which consists of an asymmetric encoder-decoder structure. The pipeline operates in two distinct phases:

  1. Joint Training (Offline): The encoder, a lightweight decoder, and the classification head are trained together on clean source data. The network optimizes a dual objective: a standard cross-entropy loss for classification and a self-supervised $l_2$ Chamfer distance loss for reconstructing heavily masked point clouds.
  2. Test-Time Training (Online): When an unlabelled, corrupted test sample arrives, MATE tokenizes the point cloud and randomly masks a massive portion of it (typically 90% to 95%). The encoder processes only the remaining visible tokens, and the decoder attempts to reconstruct the full point cloud. The encoder's weights are updated via backpropagation using this reconstruction loss before the updated model makes a final classification prediction.

The authors propose two deployment variants: MATE-Standard (which optimizes for 20 gradient steps on a single sample and resets weights afterward) and MATE-Online (which continually accumulates updates across a stream of data using a single gradient step per sample).

Figure 2. Overview of our 3D Test-Time Training methodology. We build on top of PointMAE. The input point cloud is first tokenized and then randomly masked. For our setup, we mask 90% of the point cloud. For joint training the visible tokens from the training data are fed to the encoder to get the latent embeddings from the visible tokens. These embeddings are fed to the classification head for the classification loss and concatenated with the masked tokens and fed to the decoder for reconstruction to obtain the reconstruction loss. Both losses are optimized jointly. For adaptation to an out-of-distribution test sample at test-time, we only use the MAE reconstruction task. Finally, after adapting the encoder on this single sample, evaluation is performed by using the updated encoder weights.

Datasets

To evaluate robustness against distribution shifts, the authors applied 15 common real-world corruptions (categorized into noise, transformations, and density shifts) to three standard benchmarks:

  • ModelNet-40C: A robustness benchmark derived from the ModelNet-40 dataset.
  • ShapeNet-C: A large-scale dataset created by the authors by inducing the same 15 corruptions onto the ShapeNetCore-v2 test set.
  • ScanObjectNN-C: A corrupted version of real-world scanned objects, providing a challenging testbed for practical transferability.

Key Insights and Ideas

  • Point Clouds Thrive on Extreme Masking: In 2D images, masking ratios are typically capped around 70–75%. However, because 3D point clouds are inherently sparse, MATE can mask up to 95% of the tokens during test-time training and still provide an exceptionally strong geometric signal for adaptation.
  • Radical Computational Savings: Since the transformer encoder only processes unmasked tokens, a 95% masking ratio drastically reduces the computational footprint of the forward and backward passes during test-time adaptation.
  • Reconstruction Directly Correlates with Classification: The authors empirically demonstrate that as the model's test-time reconstruction loss decreases, its classification accuracy increases. This confirms that mastering the underlying 3D geometry directly translates to better semantic understanding.
  • Sparse Adaptation Enables Real-Time Speeds: Instead of adapting on every single incoming frame, MATE can utilize an adaptation "stride" (e.g., adapting only every 5th or 50th sample). This maintains high accuracy while drastically boosting throughput to meet real-time constraints.

Key Results

  • Substantial Accuracy Gains: On ModelNet-40C, MATE-Online achieved a mean top-1 accuracy of 71.3%, a massive leap over the unadapted Source-Only baseline (53.9%). It also comfortably outperformed popular 2D TTT methods like TENT (26.5%) and SHOT (26.6%), which suffer from trivial solutions when adapted on single samples.
  • Consistent Benchmark Performance: MATE-Online achieved 69.1% accuracy on ShapeNet-C (vs. 57.6% baseline) and 48.5% on ScanObjectNN-C (vs. 45.7% baseline).
  • Real-Time Feasibility: When evaluated with an adaptation stride of 5 on an NVIDIA 3090 GPU, MATE achieved a processing speed of 20 FPS with only a minor ~3% drop in accuracy compared to per-sample adaptation. Increasing the stride further allowed throughput to reach up to 62 FPS.
Figure 3. MATE can achieve real-time adaptation performance with only a minor performance penalty. Here, we report the Mean Top-1 Accuracy (%) over the 15 corruptions in the ShapeNet-C dataset for different adaptation strides. Strides represent the number of samples after which an adaptation step is performed.

Limitations and Future Work

The primary limitation acknowledged by the authors is that MATE is currently tailored and evaluated strictly for 3D object classification. While it proves that masked autoencoders provide an excellent self-supervised signal for test-time adaptation, extending this framework to more complex downstream 3D tasks—such as 3D object detection or semantic segmentation—remains an open challenge left for future exploration.

Summary updated

Abstract

Our MATE is the first Test-Time-Training (TTT) method designed for 3D data, which makes deep networks trained for point cloud classification robust to distribution shifts occurring in test data. Like existing TTT methods from the 2D image domain, MATE also leverages test data for adaptation. Its test-time objective is that of a Masked Autoencoder: a large portion of each test point cloud is removed before it is fed to the network, tasked with reconstructing the full point cloud. Once the network is updated, it is used to classify the point cloud. We test MATE on several 3D object classification datasets and show that it significantly improves robustness of deep networks to several types of corruptions commonly occurring in 3D point clouds. We show that MATE is very efficient in terms of the fraction of points it needs for the adaptation. It can effectively adapt given as few as 5% of tokens of each test sample, making it extremely lightweight. Our experiments show that MATE also achieves competitive performance by adapting sparsely on the test data, which further reduces its computational overhead, making it ideal for real-time applications.
BibTeX
@inproceedings{mirza2022mate,
  author = {M. Jehanzeb Mirza and Inkyu Shin and Wei Lin and Andreas Schriebl and Kunyang Sun and Jaesung Choe and Horst Possegger and Mateusz Kozinski and In So Kweon and Kun-Jin Yoon and Horst Bischof},
  title = {{MATE: Masked Autoencoders are Online 3D Test-Time Learners}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2022},
}
Figure