Abstract
Coordinate-based neural representations have shown significant promise as an alternative to discrete, array-based representations for complex low dimensional signals. However, optimizing a coordinate-based network from randomly initialized weights for each new signal is inefficient. We propose applying standard meta-learning algorithms to learn the initial weight parameters for these fully-connected networks based on the underlying class of signals being represented (e.g., images of faces or 3D models of chairs). Despite requiring only a minor change in implementation, using these learned initial weights enables faster convergence during optimization and can serve as a strong prior over the signal class being modeled, resulting in better generalization when only partial observations of a given signal are available. We explore these benefits across a variety of tasks, including representing 2D images, reconstructing CT scans, and recovering 3D shapes and scenes from 2D image observations.
Introduction
Coordinate-based neural representations—where a Multi-Layer Perceptron (MLP) maps a coordinate (such as $x, y, z$) to a value (such as color or density)—have revolutionized how we represent 3D shapes and scenes, most notably in frameworks like Neural Radiance Fields (NeRF). However, these models face a major efficiency bottleneck: every time a new signal needs to be represented, a fresh optimization problem must be solved starting from random weights.
This "start from scratch" approach is problematic for two primary reasons. First, it is computationally expensive, often requiring thousands of iterations to fit a single signal. Second, a randomly initialized network lacks any inherent statistical prior. If the network is only provided with partial data—such as a few pixels of an image or a single viewpoint of a 3D object—it cannot infer missing information because it has no knowledge of the typical structure of the signal class it is modeling.
Methodology
To address these limitations, we propose using optimization-based meta-learning to find a "smart" initialization ($\theta_0$). Instead of starting from random values, the goal is to find initial weights that are just a few gradient steps away from any signal within a specific category, such as human faces or furniture.
We employ two primary meta-learning algorithms:
- Model-Agnostic Meta-Learning (MAML): An outer loop updates the initialization by evaluating how well the inner loop (standard gradient descent) performs across a variety of tasks.
- Reptile: A first-order meta-learning algorithm that is more memory-efficient than MAML, allowing for more inner-loop steps, which is essential for complex 3D tasks.
The underlying networks are standard coordinate-based MLPs, utilizing techniques like SIREN (sine activations) or Fourier features to capture high-frequency details like sharp edges and textures. By meta-learning the initialization, we essentially "bake in" the common structures of a class into the starting weights.
Key Insights and Results
Initialization as a Statistical Prior
A meta-learned initialization acts as a powerful statistical prior. For example, a face-prior initialization already resembles a blurry average face. This provides a guiding structure that allows the model to generalize from sparse data. In 2D image regression, the meta-learned initialization reached high-quality reconstruction (30.37 dB PSNR) in just two steps, whereas standard random initialization required 10 to 20 times as many iterations to reach comparable quality.
Solving the Sparse Data Problem
One of the most significant benefits of this approach is its ability to handle partial observations. Standard coordinate-based models often fail to reconstruct 3D shapes from a single photo because they lack depth cues. Because our meta-learned model "knows" the typical geometry of objects like chairs or cars, it can successfully infer the 3D structure of the unseen parts of an object from a single 2D image.
In medical imaging, specifically CT reconstruction, the meta-learned model using only two views outperformed a standard model using eight views. This demonstrates that the learned initialization provides a "common sense" baseline that significantly reduces the amount of data required for accurate reconstruction.
Conclusion and Future Work
By shifting from random to learned initializations, we significantly improve the efficiency and generalization of coordinate-based neural representations. While the method currently requires large, curated datasets of similar signals to learn a useful prior, it bridges the gap between slow, per-signal optimization and instantaneous but less flexible feed-forward architectures.
Future research may explore more sophisticated meta-learning algorithms or methods to better characterize the geometry of the weight space, potentially leading to even more robust priors that can generalize across broader and more diverse signal distributions.
Summary updated