Learned Initializations for Optimizing Coordinate-Based Neural Representations

Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, Ren Ng

Abstract

Coordinate-based neural representations have shown significant promise as an alternative to discrete, array-based representations for complex low dimensional signals. However, optimizing a coordinate-based network from randomly initialized weights for each new signal is inefficient. We propose applying standard meta-learning algorithms to learn the initial weight parameters for these fully-connected networks based on the underlying class of signals being represented (e.g., images of faces or 3D models of chairs). Despite requiring only a minor change in implementation, using these learned initial weights enables faster convergence during optimization and can serve as a strong prior over the signal class being modeled, resulting in better generalization when only partial observations of a given signal are available. We explore these benefits across a variety of tasks, including representing 2D images, reconstructing CT scans, and recovering 3D shapes and scenes from 2D image observations.

Introduction

Coordinate-based neural representations—typically Multi-Layer Perceptrons (MLPs) that map coordinates $(x, y, z)$ to values like color or density—have revolutionized the storage and rendering of 3D shapes and scenes. Unlike traditional pixels or voxels, these representations are continuous and resolution-independent. However, they suffer from a significant "cold start" problem: representing a new signal requires initializing the MLP with random weights and performing thousands of gradient descent iterations. This process is computationally expensive and ignores the structural commonalities shared within classes of signals, such as human faces or furniture.

Methods: Meta-Learning for Initialization

To address the inefficiency of random initialization, we propose using optimization-based meta-learning to find a "smart" starting point ($\theta_0^*$) for the network weights. By meta-training on a specific class of signals, the network begins optimization with weights already "primed" for that domain.

Conceptual diagram showing the coordinate-based MLP architecture and the meta-learning process for finding an optimal initialization point compared to standard initialization.

We employ two primary algorithms:

  1. Model-Agnostic Meta-Learning (MAML): An inner loop optimizes for a specific signal, while an outer loop updates the initialization to minimize post-optimization loss.
  2. Reptile: A more memory-efficient first-order alternative to MAML, allowing for more optimization steps in complex 3D tasks.

This approach requires no changes to the underlying network architecture (e.g., SIREN or ReLU with Fourier features), making it a drop-in replacement for standard initialization.

Key Insights and Results

The meta-learned initialization acts as a powerful statistical prior, capturing the underlying manifold of a signal class.

Faster Convergence

In 2D image regression tasks, the meta-learned initialization achieves high-quality reconstructions (e.g., 30.37 PSNR) in as few as 2 steps. In contrast, standard random initialization requires approximately 40 steps to reach comparable quality. This represents a significant reduction in test-time optimization costs.

Reconstruction from Sparse Data

One of the most significant advantages is the ability to reconstruct signals from partial observations. Because the initialization encodes a class-specific prior, the network can "fill in" missing information. For example, in CT imaging, high-quality reconstructions are possible with half the standard number of views.

3D reconstruction results for chairs, cars, and lamps. Meta-learned initializations enable accurate recovery from a single 2D view (SV Meta) or multiple views (MV Meta), whereas standard initialization fails to capture the geometry.

Single-View 3D Reconstruction

Using a NeRF-like model, we successfully reconstructed complex 3D shapes from a single 2D image. Standard random initialization fails at this task because it lacks the geometric prior necessary to resolve depth ambiguity. The meta-learned weights provide the structural knowledge (e.g., that a chair has four legs) required to hallucinate unseen parts accurately.

Discussion and Limitations

While meta-learned initializations offer substantial improvements in speed and generalization, they are dependent on large, curated datasets for the meta-training phase. The method's effectiveness is bounded by the diversity of the training set; it may struggle with highly unique scenes that fall outside the learned prior. Additionally, while optimization is much faster, it is not yet instantaneous, still requiring gradient descent steps at test time. Future work will explore more universal initializations across broader categories.

Summary updated

Abstract

Coordinate-based neural representations have shown significant promise as an alternative to discrete, array-based representations for complex low dimensional signals. However, optimizing a coordinate-based network from randomly initialized weights for each new signal is inefficient. We propose applying standard meta-learning algorithms to learn the initial weight parameters for these fully-connected networks based on the underlying class of signals being represented (e.g., images of faces or 3D models of chairs). Despite requiring only a minor change in implementation, using these learned initial weights enables faster convergence during optimization and can serve as a strong prior over the signal class being modeled, resulting in better generalization when only partial observations of a given signal are available. We explore these benefits across a variety of tasks, including representing 2D images, reconstructing CT scans, and recovering 3D shapes and scenes from 2D image observations.
BibTeX
@inproceedings{tancik2021learned,
  author = {Matthew Tancik and Ben Mildenhall and Terrance Wang and Divi Schmidt and Pratul P. Srinivasan and Jonathan T. Barron and Ren Ng},
  title = {{Learned Initializations for Optimizing Coordinate-Based Neural Representations}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=mP5UXv0WMpF},
}
Figure