Sign-Agnostic Implicit Learning of Surface Self-Similarities for Shape Modeling and Reconstruction from Raw Point Clouds

Wenbin Zhao, Jiabao Lei, Yuxin Wen, Jianguo Zhang, Kui Jia

Introduction

Reconstructing high-quality 3D surfaces from "raw" point clouds—data that is often noisy, incomplete, and lacks orientation (normals)—remains a fundamental challenge in computer vision. Traditional methods like Poisson Reconstruction often require accurate surface normals, while data-driven approaches may fail to generalize beyond their training sets. SAIL-S3 (Sign-Agnostic Implicit Learning of Surface Self-Similarities) addresses these issues by leveraging the universal phenomenon of surface self-similarity. By using sign-agnostic implicit learning, the framework models complex geometries directly from input data without requiring massive external datasets.

Problem Statement: The "Raw" Point Cloud Challenge

Converting discrete 3D points into a smooth, continuous surface typically falls into two categories:

  1. Classical Methods: Relying on analytic priors like smoothness, these methods (e.g., Screened Poisson Surface Reconstruction) often require precise normals. Without them, or in the presence of noise, reconstruction quality degrades significantly.
  2. Data-Driven Methods: These learn geometric priors from large datasets like ShapeNet. However, they often struggle with out-of-distribution shapes and can be memory-intensive due to global representations.

SAIL-S3 targets the reconstruction of watertight, detailed surfaces from raw point clouds that lack normals, contain noise, and have no relevant external training data.

Methodology: SAIL-S3 Framework

The core of SAIL-S3 is a local implicit modeling approach that partitions space into an adaptive set of overlapping "subfields." Each subfield handles a small patch of the surface.

Overview of the SAIL-S3 architecture, showing adaptive covering, transformed subfields, and the shared MLP producing a Signed Distance Field (SDF).

Key Components

  • Local Implicit Modeling: Space is divided into overlapping subfields, allowing for high-resolution local detail.
  • Leveraging Self-Similarity: Local geometric patterns often repeat across an object. SAIL-S3 uses a shared Multi-Layer Perceptron (MLP) for all subfields. A nuclear norm penalty on latent codes encourages the model to exploit these repeating patterns, effectively "filling in" missing data.
  • Sign-Agnostic Learning (SAL): To handle the absence of normals, the network finds a signed distance field (SDF) using only "unsigned" distance signals (distance to the nearest point).
  • Global Post-Optimization: To ensure consistency across independent local patches, a Minimum Spanning Tree (MST) algorithm is used to flip signs globally, resulting in a watertight mesh.
  • Weighted Interpolation: During inference, a specialized weighted average blends overlapping subfields to prevent "blocky" artifacts.

Key Insights: The Internal Prior

The central innovation of SAIL-S3 is the use of an "internal" prior. A single shape often contains enough redundant geometric information to regularize its own reconstruction. By treating self-similarity as a prior, the model avoids the need for 10,000 external examples to understand how to reconstruct a specific feature.

Furthermore, the authors emphasize that initialization matters. A "sphere-fitting" initialization—where local subfields start as simple spheres covering local points—is crucial for the sign-agnostic optimization to converge to a valid SDF rather than a trivial zero solution.

Evaluation and Results

The method was evaluated on ShapeNet (complex chair models) and ThreeDScans (high-resolution sculptures). It was also tested against noisy variants to assess robustness.

Qualitative comparison of SAIL-S3 against state-of-the-art methods on ShapeNet and sculpture datasets, highlighting superior detail in thin structures and sharp features.

Performance Highlights

  • Quantitative Superiority: SAIL-S3 consistently achieved the lowest Chamfer Distance (CD) and highest Normal Consistency (NC) and F-scores. On ThreeDScans, it maintained a CD of 0.003 even as other methods failed.
  • Robustness to Noise: Unlike methods requiring normal estimation (e.g., SPSR or IGR), SAIL-S3 remained stable under Gaussian noise.
  • Visual Fidelity: The method excels at reconstructing thin structures, such as chair legs, and intricate sculpture details that competitors often "blob" or fail to connect.

Limitations and Future Work

While powerful, SAIL-S3 involves "test-time optimization," making it slower than feed-forward networks (approx. 70ms per iteration over 40,000 iterations). Managing numerous local subfields and the MST-based sign flipping also adds computational overhead. Future research may focus on accelerating this optimization or handling extremely sparse point clouds where self-similarity is the primary available cue.

Summary updated

BibTeX
@inproceedings{zhao2021sign,
  author = {Wenbin Zhao and Jiabao Lei and Yuxin Wen and Jianguo Zhang and Kui Jia},
  title = {{Sign-Agnostic Implicit Learning of Surface Self-Similarities for Shape Modeling and Reconstruction from Raw Point Clouds}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=B5bprTh_eq},
}
Figure