Introduction
Implicit Neural Representations (INRs) have revolutionized 3D scene modeling and are now being applied to 2D video through Neural Representations for Videos (NeRV). Unlike traditional video formats that store pixel grids, NeRV represents a video as a continuous function—a neural network that maps a temporal coordinate (frame index) to an RGB image. While elegant, NeRV suffers from a significant practical limitation: the "from-scratch" training requirement. Every new video necessitates a full optimization process from random initialization, leading to high computational costs and slow deployment.
MetaNeRV addresses this bottleneck by leveraging meta-learning to find a robust weight initialization that facilitates rapid adaptation to unseen videos. By combining this initialization with novel spatial and temporal guidance mechanisms, MetaNeRV achieves up to a 9x speedup in convergence while maintaining or exceeding state-of-the-art performance in video compression and denoising.
The MetaNeRV Framework
The core of MetaNeRV is the application of Model-Agnostic Meta-Learning (MAML). Instead of starting from random weights, the model learns a "meta-prior"—a set of weights optimized across a diverse video corpus. When presented with a new video, the network starts from this prior, requiring only a few gradient steps to converge.
To optimize this meta-learning process for the complexities of video data, the authors introduce two key guidance strategies:
- Spatial Guidance (Multi-Resolution Loss): Standard NeRV architectures use progressive upsampling to generate high-resolution frames. MetaNeRV augments each intermediate upsampling block with a "header" that outputs a downsampled version of the target frame. By applying loss functions at multiple resolutions simultaneously, the model receives denser gradient signals, ensuring that early layers learn meaningful hierarchical features rather than becoming a bottleneck for the final output.
- Temporal Guidance (Progressive Training): Learning a meta-prior for entire video sequences is computationally difficult. MetaNeRV employs a curriculum-based temporal strategy, initially training the meta-learner to fit a single frame per video. As training progresses, the number of frames is gradually increased, allowing the model to incrementally learn temporal redundancies and motion patterns without being overwhelmed by complexity early on.
Experimental Results and Key Insights
The researchers evaluated MetaNeRV across diverse domains, including action recognition datasets (UCF101, HMDB-51), general video benchmarks (UVG), and medical imaging (EchoNet-LVH).
Convergence Speed and Performance
The results demonstrate a dramatic reduction in training time. MetaNeRV achieves a 9x speedup in convergence compared to standard NeRV. On the HMDB-51 dataset, the model outperformed baselines by approximately +6 PSNR after just a single gradient step.
The performance in medical ultrasound imaging was particularly noteworthy. Due to the consistent visual structure of ultrasound scans (e.g., the fixed scanning sector), the meta-learned weights effectively capture a structural template. On the EchoNet-LVH dataset, MetaNeRV achieved a staggering +16 PSNR improvement in one step compared to random initialization.
Downstream Applications
Beyond speed, MetaNeRV proves highly effective for downstream tasks. In video compression benchmarks, MetaNeRV combined with quantization and entropy coding outperformed traditional codecs like H.264 and HEVC, as well as existing NeRV variants, across various bitrates. The model also showed strong Out-of-Distribution (OOD) generalization; weights trained on HMDB-51 generalized remarkably well to completely unseen datasets like HOLLYWOOD2 and SVW, significantly reducing the time required to reach high-quality reconstruction thresholds.
Limitations and Future Directions
MetaNeRV demonstrates that the primary obstacle to the widespread adoption of video INRs—training latency—can be overcome through intelligent initialization. However, some constraints remain. The current model is tied to the spatial resolution it was trained on, and the meta-learning framework requires a sufficiently large dataset to learn a robust prior. Future research into resolution-agnostic architectures and improved few-shot generalization for smaller datasets promises to further expand the utility of this framework.
Summary updated