POC-SLT: Partial Object Completion with SDF Latent Transformers

Faezeh S Zakeri, Raphael Braun, Lukas Ruppert, Hendrik LenschOpenReview 2025 · Submission to ICLR 2025 · Rejected · OpenReview

TL;DR

POC-SLT introduces a blazing-fast, high-resolution method for completing partial 3D shapes using Signed Distance Fields (SDFs). By breaking 3D volumes into patches, compressing them into a smooth latent space via a Variational Autoencoder (VAE), and using a Masked Autoencoder (MAE) transformer to fill in the blanks, it completes 3D objects in a single forward pass. This approach achieves state-of-the-art accuracy and operates in mere milliseconds, making it highly practical for real-world 3D scanning and robotics applications.

What Problem Does This Paper Address?

When optical sensors (like LiDAR or depth cameras) scan the real world, they only capture the surfaces visible from their specific vantage points. This leaves massive gaps in the resulting 3D data—you might see the front of a couch, but not the back or the floor underneath it. For autonomous agents, robots, or even 3D artists to interact with these objects, those missing geometries need to be hallucinated plausibly.

Existing 3D shape completion methods often rely on point clouds or low-resolution voxels, which struggle to produce watertight, highly detailed surfaces. Recent methods that do use continuous representations like Signed Distance Fields (SDFs) often rely on autoregressive transformers. These models generate missing parts step-by-step (token-by-token), which is computationally expensive and painfully slow for high-resolution grids. POC-SLT tackles the need for a fast, high-resolution, and watertight 3D shape completion pipeline.

Methods

The authors propose POC-SLT (Partial Object Completion with SDF Latent Transformers), which shifts the heavy lifting into a compressed latent space.

Overview of the POC-SLT pipeline

The pipeline has three main steps:

  1. Patching & Encoding (P-VAE): Instead of processing a massive $128^3$ SDF grid all at once, the volume is divided into a $4 \times 4 \times 4$ grid of smaller $32^3$ patches. A custom Patch Variational Autoencoder (P-VAE) compresses each 3D patch into a compact latent vector.
  2. Single-Step Completion (SDF-Latent-Transformer): The 64 latent vectors are treated as a sequence. Patches containing missing geometry are replaced with a trainable "mask" token. A standard Transformer Encoder—trained as a Masked Autoencoder (MAE)—looks at the known patches and predicts the latent codes for all masked patches simultaneously in a single forward pass.
  3. Decoding: The completed sequence of latent codes is passed back through the P-VAE decoder to generate the final $128^3$ SDF volume, which can be easily converted into a watertight 3D mesh using Marching Cubes.

Datasets

  • ShapeNetCoreV1: The primary dataset used for training and evaluation, utilizing all 55 categories (~50k meshes) to build a strong, generalizable geometric prior.
  • ABC Dataset: A massive dataset of CAD models. A 100k subset was used to fine-tune and test the model's ability to handle mechanical and highly symmetric parts.
  • Objaverse: Used strictly for zero-shot evaluation to test how well the model generalizes to wildly out-of-distribution objects (like animals or mushrooms).
  • 3D-EPN: Used to test the model's robustness against noisy, single-view partial SDFs.

Key Insights and Ideas

  • Single-Step Inference is King: By framing shape completion as a Masked Autoencoder task rather than an autoregressive generation task, the model predicts all missing geometry in parallel. This completely bypasses the sequential bottleneck of previous state-of-the-art models.
  • Patch-Based Processing Scales Beautifully: Breaking the volume into $32^3$ patches strikes a perfect balance. The patches are small enough that the network learns generalizable local geometric features (like corners, edges, or flat surfaces) across different object classes, but large enough to capture meaningful context.
  • Smooth Latent Spaces Beat Quantization: Unlike competing methods (e.g., AutoSDF) that use Vector Quantized VAEs (VQ-VAEs), POC-SLT uses a standard VAE with KL regularization. This avoids discrete quantization errors and preserves much finer geometric details in the final mesh.

Key Results

  • Unmatched Speed: Because of the single-step transformer, POC-SLT completes shapes in just 8.6 milliseconds. For comparison, AutoSDF takes 4.3 seconds, and ShapeFormer takes 20 seconds.
  • Higher Resolution: The method comfortably outputs $128^3$ resolution SDFs on consumer-grade GPUs, whereas previous latent SDF methods were largely capped at $64^3$.
  • State-of-the-Art Accuracy: On the ShapeNet completion task (where only the bottom half of the object is provided), POC-SLT significantly outperformed baselines. It achieved a Unidirectional Hausdorff Distance (UHD) of 0.0354 and a Chamfer Distance (CD) of 3.35, compared to AutoSDF's 0.0618 and 6.18, respectively.
Qualitative comparison on ShapeNet
  • Strong Generalization: The model successfully hallucinated plausible completions for out-of-distribution Objaverse objects (like rocking chairs and animals) that it had never seen during training.

Limitations and Future Work

  • Fixed Bounding Boxes: The model currently assumes objects fit within a normalized, fixed bounding box. This is fine for isolated objects but limits its direct application to unbounded, open-world scenes. The authors suggest future work could explore variable-length sequences to handle larger spatial encodings.
  • Binary Patch Knowledge: The transformer assumes a patch is either 100% known or 100% masked. If a patch contains only partial surface data (e.g., a noisy single-view scan), the model struggles to repair it and just feeds the corrupt input through. Future iterations could benefit from a mechanism that accepts partially filled patches as "hints" rather than discarding or blindly trusting them.

Summary updated

Abstract

3D geometric shape completion hinges on representation learning and a deep understanding of geometric data. Without profound insights into the three-dimensional nature of the data, this task remains unattainable. Our work addresses this challenge of 3D shape completion given partial observations by proposing a transformer operating on the latent space representing Signed Distance Fields (SDFs). Instead of a monolithic volume, the SDF of an object is partitioned into smaller high-resolution patches leading to a sequence of latent codes. The approach relies on a smooth latent space encoding learned via a variational autoencoder (VAE), trained on millions of 3D patches. We employ an efficient masked autoencoder transformer to complete partial sequences into comprehensive shapes in latent space. Our approach is extensively evaluated on partial observations from ShapeNet and the ABC dataset where only fractions of the objects are given. The proposed POC-SLT architecture compares favorably with several baseline state-of-the-art methods, demonstrating a significant improvement in 3D shape completion, both qualitatively and quantitatively.
BibTeX
@misc{zakeri2025poc,
  author = {Faezeh S Zakeri and Raphael Braun and Lukas Ruppert and Hendrik Lensch},
  title = {{POC-SLT: Partial Object Completion with SDF Latent Transformers}},
  year = {2025},
  url = {https://openreview.net/forum?id=aSByBbmASe},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure