nerf2nerf: Pairwise Registration of Neural Radiance Fields

Lily Goli, Daniel Rebain, Sara Sabour, Animesh Garg, Andrea Tagliasacchi

Introduction

3D registration—the process of aligning two different 3D captures of the same object or scene—is a cornerstone of computer vision and robotics. Traditionally, this is performed using "Lagrangian" representations like point clouds or meshes, where corresponding points are identified to minimize spatial distance, often via the Iterative Closest Point (ICP) algorithm.

However, Neural Radiance Fields (NeRFs) present a challenge as "Eulerian" or field-based representations. They lack explicit points, existing instead as continuous functions. Standard registration typically requires extracting noisy point clouds, which sacrifices the precision of the continuous neural representation. Furthermore, NeRFs encode view-dependent lighting and color (radiance). Attempting to align two NeRFs of the same scene captured under different lighting conditions (e.g., noon vs. sunset) using direct color comparison is prone to failure.

Methods

The authors propose nerf2nerf, a robust optimization framework that operates directly on neural fields. The pipeline transitions from raw image captures to a high-precision optimized registration.

The nerf2nerf pipeline: from input images and NeRF training to keypoint-guided optimized registration.

The Surface Field ($S$)

A key innovation is the distillation of a "surface field" from the NeRF’s density field. Unlike radiance, which is sensitive to lighting changes, the surface field measures the likelihood of a point residing on an object's surface. This provides a lighting-invariant geometric signal crucial for accurate registration.

Distilling the surface field: The method extracts a clean geometric signal from the noisy density field, making it robust to lighting variations.

Optimization Framework

To facilitate registration, the framework employs several core components:

  • Differentiable Matching: The categorical "surface vs. empty space" field is convolved with a Gaussian to create a smooth "receptive field." This allows gradients to flow, enabling standard optimizers to solve for the best rigid transformation ($T$).
  • Hybrid Energy Function: The optimization minimizes a combination of $L_{match}$ (aligning surface fields) and $L_{key}$ (using sparse, human-annotated keypoints for initial alignment and constraint).
  • Metropolis-Hastings Sampling: To manage computational costs, a smart sampling strategy focuses optimization on informative surface areas rather than empty space.

Datasets and Evaluation

The method was evaluated across diverse scenarios:

  • Synthetic Dataset: 8 scene pairs rendered in Blender (PolyHaven and NeRF-Blender models) for quantitative benchmarking against ground truth.
  • Real-World Dataset: 6 "in-the-wild" scene pairs reconstructed using mip-NeRF 360, testing robustness against real-world noise.
  • Partial Overlap Benchmark: A test case involving a "Max Planck" bust with separate captures of the left and right sides to evaluate the fusion of incomplete models.

Key Results

nerf2nerf demonstrates significant improvements over traditional methods. On synthetic data, it achieved nearly 18x lower 3D-ADD error in the "elephant" scene compared to Fast Global Registration (FGR) applied to extracted points. Qualitatively, the results show perfect "z-fighting" between aligned scenes, indicating precise surface overlap. The method also successfully "stitched" partial captures into complete 3D models with minimal shared geometry.

Discussion and Limitations

By prioritizing geometry over radiance, nerf2nerf leverages the continuous nature of fields to achieve higher precision than point-based algorithms. However, challenges remain:

  • Geometric Ambiguity: Small objects on large flat surfaces can cause the sampler to converge on the table's geometry instead of the object. Incorporating lighting-invariant color gradients may resolve this.
  • Manual Initialization: While only a few keypoints are needed, the process currently requires human intervention. Future iterations could automate this via deep feature matching.
  • Rigid Constraints: The current framework is limited to rigid transformations (rotation and translation), leaving non-rigid registration (e.g., human poses) as a future research direction.

Summary updated

BibTeX
@inproceedings{goli2023nerf,
  author = {Lily Goli and Daniel Rebain and Sara Sabour and Animesh Garg and Andrea Tagliasacchi},
  title = {{nerf2nerf: Pairwise Registration of Neural Radiance Fields}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160794},
}
Figure