Abstract
NeRV-Diffusion is a novel video synthesis framework that generates videos by using a diffusion model to synthesize the weights of an Implicit Neural Representation (INR). By shifting the generation process from pixel/feature space to "neural weight space," the model achieves high-quality, temporally consistent video synthesis with a significantly smaller memory footprint and faster decoding than traditional video diffusion models.
Introduction
Current state-of-the-art video generation typically relies on Video Latent Diffusion Models (LDMs), which encode videos as sequences of individual frame-wise feature maps. This approach suffers from several drawbacks: high redundancy by ignoring inter-frame coherence, extreme computational costs due to heavy cross-frame attention layers, and slow decoding times from large Variational Autoencoders (VAEs).
NeRV-Diffusion addresses these issues by leveraging Implicit Neural Representations (INRs)—networks where weights themselves store visual information. While INRs are efficient, their weights are typically not "well-behaved" for standard diffusion models. NeRV-Diffusion introduces a method to make these neural weights "diffusable," enabling efficient video generation in weight space.
Methodology
The NeRV-Diffusion pipeline consists of three primary components designed to bridge the gap between INRs and diffusion models:
- The NeRV Autoencoder (Tokenizer): A hypernetwork-based encoder compresses raw video into "latent weight tokens." A KL Bottleneck forces these tokens toward a Gaussian distribution, while Channel-wise Parameterization allows tokens to directly define the convolution kernels of the decoder.
- The Implicit Diffusion Transformer (Implicit DiT): This generative engine operates entirely in weight space, iteratively denoising random noise to produce final INR weights.
- The Generative NeRV Decoder: An efficient convolutional architecture that uses spatiotemporal embeddings to decode the generated weights into a full video.
Because the video is represented as a single set of weights, temporal consistency is inherently "baked into" the representation, eliminating the need for expensive temporal attention layers during denoising.
Key Insights
- Weight Synthesis as Meta-Learning: Video synthesis is treated as generating a function (the weights) rather than a grid of pixels.
- Holistic Compression: The INR captures the "keyframe-residue" relationship naturally, avoiding the need for explicit motion vectors or optical flow.
- Latent Space Smoothness: The KL bottleneck enables a smooth latent space, allowing for seamless interpolation between different video weights.
- Efficiency: The model achieves high performance with a fraction of the parameters used by models like MAGVIT; for instance, the "Small" version uses only 3.5M parameters for the detokenizer.
Experimental Results
NeRV-Diffusion was evaluated on standard benchmarks including UCF-101, Kinetics-600, and TaiChi-HD.
Key findings include:
- State-of-the-Art Quality: NeRV-Diffusion-L achieved a gFVD of 97 on UCF-101, outperforming previous INR-based models like DIGAN and remaining competitive with much larger non-implicit models.
- Compact Representation: The framework uses only 128 tokens to represent a video, compared to 1024+ tokens in models like TATS or MAGVIT.
- Temporal Flexibility: The continuous time coordinate input allows for 8x interpolation (e.g., turning a 16-frame video into 128 frames) without retraining.
Conclusion and Future Work
NeRV-Diffusion demonstrates that generating neural weights is a viable and efficient alternative to pixel-space video synthesis. While current scaling is limited to $128 \times 128$ resolution, future work aims to address high-definition scaling and complex text-to-video conditioning. This "weight-generation" paradigm also holds promise for 3D scene synthesis applications.
Summary updated