Introduction
Implicit Neural Representations (INRs)—which use coordinate-based Multilayer Perceptrons (MLPs) to represent signals like images, 3D shapes, or radiance fields—have become a cornerstone of modern computer vision. They are highly advantageous for fitting complex details and solving inverse problems compared to discrete grid-based representations.
However, this paper addresses a major practical bottleneck: tedious per-scene training. To represent a single new image or 3D object, you typically have to train a network from scratch on dense measurements. This is computationally expensive and frequently fails when data is sparse. To solve this, the authors introduce the Neural Implicit Dictionary (NID), a framework that replaces the slow, data-hungry process of training individual INRs with a "dictionary" of pre-learned neural function bases. By representing new scenes as sparse combinations of these experts via a Mixture-of-Experts (MoE) architecture, NID achieves up to 100× faster convergence and can reconstruct high-fidelity signals using 98% less data than traditional methods.
The Bottleneck of Traditional INRs
While meta-learning (learning a good starting point for weights) and feature-grid methods have tried to solve the per-scene training bottleneck, they often struggle with high-frequency details or require significant supervision. The authors identify that reconstructing an INR from few-shot measurements is essentially an underdetermined system.
Looking to the classical field of compressed sensing for a solution, they hypothesized that if we can learn a "dictionary" of representative functions, we can represent any new scene by simply finding the right "code" to combine them. NID successfully translates this "sparse dictionary" philosophy from discrete signal processing to the continuous neural domain.
Methodology: The Neural Implicit Dictionary
The core innovation is replacing a single large network with a group of small coordinate-based subnetworks (experts) that act as a continuous function basis.
- Functional Combination: A signal $f(x)$ is represented as $f(x) = \sum_{i=1}^{n} \alpha_i b_{\theta_i}(x)$, where $b_{\theta_i}$ are the neural experts and $\alpha_i$ are sparse coding coefficients.
- Mixture-of-Experts (MoE) Training: To handle thousands of potential experts efficiently, the authors use a sparse gating mechanism. For any given input, only a handful of experts are activated, saving massive computational overhead.
- Architecture Efficiency: To keep the model compact, all experts share a 4-layer backbone (positional embedding and early features), with only the final two layers being independent for each expert.
- Two-Stage Workflow:
- Offline: The dictionary is trained on a large collection of data (e.g., thousands of faces or 3D chairs) to span the desired function space.
- Online: For a new, unseen scene, the dictionary weights are frozen. The model only needs to solve for the sparse coding coefficients $\alpha$, which is significantly faster than full network training.
Unlike traditional INRs that often use data-agnostic sinusoidal bases (like SIREN) prone to overfitting, NID learns its bases directly from data. This allows it to capture intrinsic domain-specific information, such as the underlying structure of a human face.
Key Results
The authors validated NID across several modalities, including 2D image regression (CelebA), self-supervised background decomposition (BMC-Real), CT reconstruction (Shepp-Logan Phantoms), and 3D geometry from point clouds (ShapeNet).
- Speed and Optimization: By only tuning the coding coefficients during inference, the model avoids the "saddle point" issues that plague meta-learning approaches. In image regression, NID achieves high-quality results 100× faster than standard training.
- Data Efficiency: NID can reconstruct 3D shapes (Signed Distance Functions) with 98% fewer point samples than baselines. In CT reconstruction, it maintains high fidelity with 90% fewer viewing angles.
- Robustness to Corruption: NID excels at inverse problems like inpainting. Because the dictionary acts as a strong prior, it can effectively "ignore" corruptions (like occluding boxes) by projecting the corrupted image onto the learned subspace of clean data.
Limitations and Future Work
While highly effective, NID has a few limitations. For very large or complex signals, a single dictionary is insufficient. The authors currently use a "patch-wise" approach, partitioning the coordinate space, but finding a seamless way to scale to massive scenes (like city-scale NeRFs) remains an open challenge. Additionally, the choice between using an encoder-based gating network versus a lookup table involves a trade-off between inference speed and convergence quality that warrants further optimization. Future work could also benefit from deeper theoretical analysis connecting NID to formal subspace learning theories.
Summary updated