TL;DR
This paper introduces STRAINER, a novel framework for training Implicit Neural Representations (INRs) that learns transferable features by sharing initial encoder layers across multiple INRs. This approach significantly accelerates convergence and improves reconstruction quality for fitting new, similar signals, effectively enabling INRs to learn data-driven priors.
What Problem Does This Paper Address?
Implicit Neural Representations (INRs) have shown great promise in various applications like inverse problems and neural rendering. However, a key limitation is that INRs are typically trained to represent a single signal, leading to highly specialized features that don't easily transfer to new signals. This lack of transferability hinders their broader applicability, especially in data-scarce scenarios or when dealing with distributions of related signals. The authors aim to bridge this gap by enabling INRs to learn generalizable features that can be effectively transferred, similar to how transfer learning works with CNNs and Transformers.
Methods
The core innovation is the STRAINER framework. It decomposes an INR into an "encoder" that maps input coordinates to features and a "decoder" that maps these features to the output signal. During training, STRAINER fits multiple INRs to signals from a specific domain, but crucially, it shares the initial encoder layers across all these INRs while using independent decoders for each signal. This shared encoder learns a rich, transferable representation of the domain's underlying structure.
At test time, a new INR for an unseen signal is initialized with the pre-trained shared encoder and a randomly initialized decoder. This initialization significantly speeds up convergence and improves reconstruction quality compared to training an INR from scratch. The authors leverage the SIREN (Sinusoidal Representation Networks) architecture as their base model.
Datasets
The authors evaluated STRAINER on several datasets:
- CelebA-HQ: Used for training and testing face images.
- Animal Faces-HQ (AFHQ): Specifically used the cat subset for training and testing.
- OASIS-MRI: Used for medical imaging experiments.
- Stanford Cars and Flowers: Used for out-of-domain generalization tests.
- Kodak: Used for high-resolution image fitting.
- ShapeNet: Used for 3D object fitting experiments.
Key Insights and Ideas
- Initial layers capture generalizable structure: The paper hypothesizes and demonstrates that the initial layers of an INR learn low-frequency features that are common across similar signals, making them ideal for transfer.
- Shared encoder as a data-driven prior: By training a shared encoder on a distribution of signals, STRAINER effectively encodes a data-driven prior into the INR's initialization.
- Faster convergence and better quality: The learned initialization dramatically speeds up the fitting process for new signals and leads to significantly higher reconstruction quality.
- Surprising out-of-domain transferability: STRAINER shows impressive performance even when tested on datasets different from its training domain, suggesting a robust feature learning capability.
- Simplicity and efficiency: Compared to complex meta-learning approaches, STRAINER offers a simpler and more computationally efficient way to achieve transferable INR features.
Key Results
- In-domain image fitting: STRAINER achieved approximately a +10dB gain in signal quality early on compared to an untrained INR. For CelebA-HQ, STRAINER-10 (trained on 10 images) achieved a PSNR of 57.80 dB, significantly outperforming vanilla SIREN (44.91 dB) and even Meta-Learned 5K (53.08 dB).
- Out-of-domain image fitting: STRAINER-10, trained on CelebA-HQ, showed surprisingly strong performance on AFHQ and OASIS-MRI, outperforming Meta-Learned 5K.
- Inverse problems (Super-resolution, Denoising): STRAINER demonstrated faster convergence and comparable or better PSNR than SIREN for super-resolution tasks. For example, STRAINER-10 (Fast) achieved comparable PSNR to SIREN in approximately a third of the runtime.
- 3D object fitting: STRAINER achieved a 12.3% relative improvement in IOU compared to random initialization for fitting a 'Thai Statue' 3D object after being pre-trained on 'Chair' objects from ShapeNet.
- Training efficiency: STRAINER's initialization learning takes only 24.54 seconds on an Nvidia A100 GPU for 5000 iterations, significantly faster than Meta-learned 5K.
Limitations and Future Work
- Occasional instability: The authors note occasional PSNR "drops" during test-time fitting due to the encoder being tuned while the decoder is random, though they state STRAINER usually recovers quickly.
- Characterizing transferred features: It's not fully clear what features are transferred, how they change across different image distributions, and how they directly compare to transfer learning in CNNs and Transformers. Further work is needed to characterize these aspects.
- Broader applicability: The authors suggest future work could explore adapting STRAINER for other INR applications like 3D occupancy maps, radiance fields, or video, and investigating its potential for patient-specific medical imaging and real-time domains.
Summary updated