Learned Initializations for Optimizing Coordinate-Based Neural Representations

Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, Ren Ng

Abstract

Coordinate-based neural representations have shown significant promise as an alternative to discrete, array-based representations for complex low dimensional signals. However, optimizing a coordinate-based network from randomly initialized weights for each new signal is inefficient. We propose applying standard meta-learning algorithms to learn the initial weight parameters for these fully-connected networks based on the underlying class of signals being represented (e.g., images of faces or 3D models of chairs). Despite requiring only a minor change in implementation, using these learned initial weights enables faster convergence during optimization and can serve as a strong prior over the signal class being modeled, resulting in better generalization when only partial observations of a given signal are available. We explore these benefits across a variety of tasks, including representing 2D images, reconstructing CT scans, and recovering 3D shapes and scenes from 2D image observations.

Introduction

Coordinate-based neural representations—where a Multi-Layer Perceptron (MLP) maps a coordinate (such as $x, y, z$) to a value (such as color or density)—have revolutionized how we represent 3D shapes and scenes, most notably in frameworks like Neural Radiance Fields (NeRF). However, these models face a major efficiency bottleneck: every time a new signal needs to be represented, a fresh optimization problem must be solved starting from random weights.

This "start from scratch" approach is problematic for two primary reasons. First, it is computationally expensive, often requiring thousands of iterations to fit a single signal. Second, a randomly initialized network lacks any inherent statistical prior. If the network is only provided with partial data—such as a few pixels of an image or a single viewpoint of a 3D object—it cannot infer missing information because it has no knowledge of the typical structure of the signal class it is modeling.

Methodology

To address these limitations, we propose using optimization-based meta-learning to find a "smart" initialization ($\theta_0$). Instead of starting from random values, the goal is to find initial weights that are just a few gradient steps away from any signal within a specific category, such as human faces or furniture.

Conceptual overview of standard vs. meta-learned initialization in weight space.

We employ two primary meta-learning algorithms:

  • Model-Agnostic Meta-Learning (MAML): An outer loop updates the initialization by evaluating how well the inner loop (standard gradient descent) performs across a variety of tasks.
  • Reptile: A first-order meta-learning algorithm that is more memory-efficient than MAML, allowing for more inner-loop steps, which is essential for complex 3D tasks.

The underlying networks are standard coordinate-based MLPs, utilizing techniques like SIREN (sine activations) or Fourier features to capture high-frequency details like sharp edges and textures. By meta-learning the initialization, we essentially "bake in" the common structures of a class into the starting weights.

Key Insights and Results

Initialization as a Statistical Prior

A meta-learned initialization acts as a powerful statistical prior. For example, a face-prior initialization already resembles a blurry average face. This provides a guiding structure that allows the model to generalize from sparse data. In 2D image regression, the meta-learned initialization reached high-quality reconstruction (30.37 dB PSNR) in just two steps, whereas standard random initialization required 10 to 20 times as many iterations to reach comparable quality.

Solving the Sparse Data Problem

One of the most significant benefits of this approach is its ability to handle partial observations. Standard coordinate-based models often fail to reconstruct 3D shapes from a single photo because they lack depth cues. Because our meta-learned model "knows" the typical geometry of objects like chairs or cars, it can successfully infer the 3D structure of the unseen parts of an object from a single 2D image.

3D reconstruction from single (SV) and multiple (MV) views using meta-learned initializations compared to standard initialization.

In medical imaging, specifically CT reconstruction, the meta-learned model using only two views outperformed a standard model using eight views. This demonstrates that the learned initialization provides a "common sense" baseline that significantly reduces the amount of data required for accurate reconstruction.

Conclusion and Future Work

By shifting from random to learned initializations, we significantly improve the efficiency and generalization of coordinate-based neural representations. While the method currently requires large, curated datasets of similar signals to learn a useful prior, it bridges the gap between slow, per-signal optimization and instantaneous but less flexible feed-forward architectures.

Future research may explore more sophisticated meta-learning algorithms or methods to better characterize the geometry of the weight space, potentially leading to even more robust priors that can generalize across broader and more diverse signal distributions.

Summary updated

Abstract

Coordinate-based neural representations have shown significant promise as an alternative to discrete, array-based representations for complex low dimensional signals. However, optimizing a coordinate-based network from randomly initialized weights for each new signal is inefficient. We propose applying standard meta-learning algorithms to learn the initial weight parameters for these fully-connected networks based on the underlying class of signals being represented (e.g., images of faces or 3D models of chairs). Despite requiring only a minor change in implementation, using these learned initial weights enables faster convergence during optimization and can serve as a strong prior over the signal class being modeled, resulting in better generalization when only partial observations of a given signal are available. We explore these benefits across a variety of tasks, including representing 2D images, reconstructing CT scans, and recovering 3D shapes and scenes from 2D image observations.
BibTeX
@inproceedings{tancik2021learned,
  author = {Matthew Tancik and Ben Mildenhall and Terrance Wang and Divi Schmidt and Pratul P. Srinivasan and Jonathan T. Barron and Ren Ng},
  title = {{Learned Initializations for Optimizing Coordinate-Based Neural Representations}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=mP5UXv0WMpF},
}

Loading related content...

Figure