Learning Transferable Features for Implicit Neural Representations

Kushal Vyas, Ahmed Imtiaz Humayun, Aniket Dashpute, Richard Baraniuk, Ashok Veeraraghavan, Guha Balakrishnan

TL;DR

STRAINER is a simple yet powerful framework that makes Implicit Neural Representations (INRs) transferable. By sharing initial "encoder" layers across a small set of signals from the same domain, it learns a data-driven prior that allows new INRs to converge significantly faster and achieve up to +10dB higher reconstruction quality than standard random initialization.

The "Cold Start" Problem of INRs

Implicit Neural Representations (INRs)—like the popular SIREN or NeRF—map coordinates (like $x, y$) to signal values (like RGB). While they are excellent at capturing continuous detail, they suffer from a major "cold start" problem: an INR is typically trained from scratch for every single signal.

Because each network is overfit to one specific image or 3D shape, the learned features are generally considered non-transferable. This makes INRs slow to train and prevents them from leveraging common patterns within a dataset (e.g., all human faces share a similar basic structure). While meta-learning approaches exist to solve this, they are often computationally expensive, unstable, and require massive amounts of training data.

Introducing STRAINER

To address this, the authors introduce STRAINER (Shared TRansferable Architecture for INRs), which rethinks the INR as a two-part system:

  1. The Encoder: The first several layers of the MLP.
  2. The Decoder: The remaining layers leading to the output.
STRAINER Framework

The Training Phase: Instead of training one network on one image, STRAINER trains $N$ networks (usually just 10) on $N$ different images from the same domain simultaneously. The key trick is that all $N$ networks share the same encoder weights but maintain independent decoders. This forces the encoder to learn a "common language" or partition space that represents the general morphology of the domain (like the layout of a face).

The Test Phase: When fitting a brand-new, unseen signal, the network is initialized with the pre-trained shared encoder and a random decoder. The entire network is then optimized as usual, but with a massive head start.

Datasets and Evaluation

The researchers evaluated STRAINER across a diverse range of signal types:

  • Natural Images: CelebA-HQ (faces) and AFHQ (cat faces).
  • Medical Imaging: OASIS-MRI (brain scans).
  • Generalization/OOD: Stanford Cars, Flowers, and the Kodak dataset for high-resolution testing.
  • 3D Data: ShapeNet (specifically the 'Chair' category) and the 'Thai Statue' for 3D occupancy fitting.

Key Insights: The "Ghost" in the Machine

By visualizing the input space partitions using PCA on the learned features, the authors found that STRAINER’s encoder develops a "ghost-like" structure of the target domain before it even sees the test image.

PCA of Learned Features
  • Structural Priors: As seen above, standard INRs (like SIREN) start with random, chaotic partitions. STRAINER, however, initializes with a recognizable base structure, allowing it to capture high-frequency details much earlier in the optimization process.
  • Minimalist Meta-Learning: Unlike traditional meta-learning (like MAML) which might require 14,000 images and hours of compute, STRAINER achieves superior results using just 10 images and about 25 seconds of training.
  • Surprising Generalization: An encoder trained on human faces (CelebA) actually provides a better initialization for MRI scans or cats than a random start. This suggests the encoder learns a general "natural signal prior" that transcends specific categories.

Key Results

  • Instant Quality Boost: At the start of test-time fitting, STRAINER provides a ~10dB PSNR gain over vanilla SIREN.
  • Speedup: In inverse problems like super-resolution and denoising, STRAINER recovers clean images 3x faster than standard baselines.
  • 3D Superiority: For 3D occupancy fitting, STRAINER achieved a 12.3% relative improvement in Intersection over Union (IoU) compared to random initialization after only 150 iterations.
  • Efficiency: STRAINER-10 (trained on 10 images) outperforms state-of-the-art transformer-based hypernetworks (like TransINR) while being significantly lighter and faster to train.

Limitations and Future Work

  • Occasional Instability: The authors noted occasional "PSNR drops" during the fitting process, likely due to the mismatch between the pre-trained encoder and the randomly initialized decoder, though the model usually recovers quickly.
  • Theoretical Gap: While the empirical results are strong, the mathematical "why" behind which specific features are transferred remains an open question.
  • Scaling: Future work could explore how these transferable features behave when scaled to much larger, more complex 3D scenes or video representations.

Summary updated

Abstract

Implicit neural representations (INRs) have demonstrated success in a variety of applications, including inverse problems and neural rendering. An INR is typically trained to capture one signal of interest, resulting in learned neural features that are highly attuned to that signal. Assumed to be less generalizable, we explore the aspect of transferability of such learned neural features for fitting similar signals. We introduce a new INR training framework, STRAINER that learns transferable features for fitting INRs to new signals from a given distribution, faster and with better reconstruction quality. Owing to the sequential layer-wise affine operations in an INR, we propose to learn transferable representations by sharing initial encoder layers across multiple INRs with independent decoder layers. At test time, the learned encoder representations are transferred as initialization for an otherwise randomly initialized INR. We find STRAINER to yield extremely powerful initialization for fitting images from the same domain and allow for a ≈ +10dB gain in signal quality early on compared to an untrained INR itself. STRAINER also provides a simple way to encode data-driven priors in INRs. We evaluate STRAINER on multiple in-domain and out-of-domain signal fitting tasks and inverse problems and further provide detailed analysis and discussion on the transferability of STRAINER’s features.
BibTeX
@inproceedings{vyas2024learning,
  author = {Kushal Vyas and Ahmed Imtiaz Humayun and Aniket Dashpute and Richard Baraniuk and Ashok Veeraraghavan and Guha Balakrishnan},
  title = {{Learning Transferable Features for Implicit Neural Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=ABYdKpDb8p},
}
Figure