PointMAC: Meta-Learned Adaptation for Robust Test-Time Point Cloud Completion

Linlian Jiang, Rui Ma, Li Gu, Ziqiang Wang, Xinxin Zuo, Yang Wang

TL;DR

PointMAC is a novel framework that allows 3D point cloud completion models to adapt to specific, unseen objects at test time. By using meta-learning and self-supervised auxiliary tasks, the model "refines" its understanding of each individual input on-the-fly, moving beyond static, "one-size-fits-all" reconstructions to achieve state-of-the-art accuracy on both synthetic and real-world data.

What Problem Does This Paper Address?

3D sensors like LiDAR are essential for robots and self-driving cars, but they often produce "incomplete" point clouds due to occlusions (something blocking the view) or sensor noise. Point cloud completion models aim to fill in these gaps.

However, current state-of-the-art models are static. They learn a set of "inductive biases" during training—essentially a memory of what a generic car or chair looks like. At test time, they apply these fixed rules to every input. This rigidity is a bottleneck: if the model encounters a unique object shape or a type of sensor noise it didn't see during training, it fails to adapt. Instead of reconstructing the specific object in front of it, it produces a "generic" version that might ignore crucial visible cues.

Comparison of static inference yielding generic completion versus PointMAC's test-time adaptation for sample-specific completion.

Methods

The authors introduce PointMAC, which shifts from static inference to dynamic, sample-specific adaptation. The core idea is to let the model "practice" on the specific input it is trying to complete before giving its final answer.

Key components include:

  • Bi-Aux Units: The model is equipped with two self-supervised auxiliary tasks:
  • Stochastic Masked Reconstruction: The model hides parts of the input and tries to recover them, forcing it to learn the object's underlying structure.
  • Artifact Denoising: The model adds synthetic noise to the input and learns to clean it, making it robust to sensor distortions.
  • Meta-Auxiliary Learning (via MAML): To ensure these practice tasks actually help the main goal, the authors use Model-Agnostic Meta-Learning (MAML). During training, the model is optimized so that a few "practice steps" on the auxiliary tasks will lead to the best possible performance on the primary completion task.
  • Adaptive $\lambda$-Calibration: Balancing multiple loss functions (tasks) is notoriously difficult. This meta-learned mechanism dynamically adjusts the importance of each auxiliary task during training to prevent them from interfering with each other.
  • Test-Time Adaptation (TTA): This is where the magic happens. When the model sees a new point cloud at inference time, it performs a few quick gradient descent steps to update its encoder based on the auxiliary tasks. The decoder remains fixed to preserve the global shape priors, while the encoder becomes "specialized" for that specific object.

Datasets

The researchers evaluated PointMAC across a diverse range of benchmarks:

  • PCN & ShapeNet-55/34: Standard synthetic datasets containing tens of thousands of CAD models used to test general reconstruction and category-specific performance.
  • MVP (Multi-View Partial): A high-fidelity simulated dataset that mimics real-world scanning conditions, including complex occlusions.
  • KITTI: A challenging real-world dataset consisting of LiDAR scans from actual autonomous driving scenarios. Since KITTI lacks "ground truth" (perfect versions of the cars), it is used to test how well models generalize from synthetic training to the messy real world.

Key Insights and Ideas

  • The "Generic" vs. "Specific" Trap: The authors identify that static models often ignore visible "cues" in favor of training priors. PointMAC breaks this by allowing the model to "look closer" at the unique geometry of the input.
  • Alignment is Everything: Simply adding auxiliary tasks can sometimes hurt performance (negative transfer). By using MAML, the authors ensure that the auxiliary tasks are mathematically "aligned" with the primary goal of completion.
  • Encoder Rigidity is the Bottleneck: The paper argues that the decoder usually knows how to build shapes; the problem is the encoder failing to extract the right features from noisy, novel inputs. Adapting only the encoder at test time is a computationally efficient way to solve this.

Key Results

PointMAC achieved state-of-the-art (SOTA) results across the board:

  • PCN Dataset: Achieved a Chamfer Distance (CD-Avg) of 6.33, outperforming the previous best (CRA-PCN at 6.39).
  • ShapeNet-55: Set a new benchmark with a CD-Avg of 0.83 and an F1 score of 0.490.
  • Real-World Generalization: On the KITTI dataset, PointMAC reduced the "Fidelity" error (how well the output matches the input) by 10.6% compared to non-adaptive methods.
  • Qualitative Superiority: Visual results show PointMAC successfully recovering fine details—like thin chair slats, airplane propellers, and delicate lamp arms—that other models either smoothed over or missed entirely.
Qualitative comparison showing PointMAC recovering fine details like airplane propellers and lamp arms better than baseline methods.

Limitations and Future Work

While PointMAC is a significant step forward, the authors note a few areas for growth:

  • Single-Object Focus: Currently, the framework is optimized for completing single objects. Extending this to "scene-level" completion (e.g., an entire street corner) is a natural next step.
  • Supervision Costs: While the test-time adaptation is label-free, the initial meta-training still requires paired ground-truth data.
  • Multi-Modal Potential: The authors suggest that future versions could incorporate other data sources, like 2D images, to provide even stronger guidance for the test-time adaptation process.

Summary updated

Abstract

Point cloud completion is essential for robust 3D perception in safety-critical applications such as robotics and augmented reality. However, existing models perform static inference and rely heavily on inductive biases learned during training, limiting their ability to adapt to novel structural patterns and sensor-induced distortions at test time. To address this limitation, we propose PointMAC, a meta-learned framework for robust test-time adaptation in point cloud completion. It enables sample-specific refinement without requiring additional supervision. Our method optimizes the completion model under two self-supervised auxiliary objectives that simulate structural and sensor-level incompleteness. A meta-auxiliary learning strategy based on Model-Agnostic Meta-Learning (MAML) ensures that adaptation driven by auxiliary objectives is consistently aligned with the primary completion task. During inference, we adapt the shared encoder on-the-fly by optimizing auxiliary losses, with the decoder kept fixed. To further stabilize adaptation, we introduce Adaptive $\lambda$-Calibration, a meta-learned mechanism for balancing gradients between primary and auxiliary objectives. Extensive experiments on synthetic, simulated, and real-world datasets demonstrate that PointMAC achieves state-of-the-art results by refining each sample individually to produce high-quality completions. To the best of our knowledge, this is the first work to apply meta-auxiliary test-time adaptation to point cloud completion.
BibTeX
@inproceedings{jiang2025pointmac,
  author = {Linlian Jiang and Rui Ma and Li Gu and Ziqiang Wang and Xinxin Zuo and Yang Wang},
  title = {{PointMAC: Meta-Learned Adaptation for Robust Test-Time Point Cloud Completion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=4xl00JWQ1z},
}
Figure