NeRV-Diffusion: Diffuse Implicit Neural Representations for Video Synthesis

Yixuan Ren, Hanyu Wang, Hao Chen, Bo He, Abhinav ShrivastavaarXiv 2025 · DOI · arXiv

Abstract

NeRV-Diffusion is a novel video synthesis framework that generates videos by using a diffusion model to synthesize the weights of an Implicit Neural Representation (INR). By shifting the generation process from pixel/feature space to "neural weight space," the model achieves high-quality, temporally consistent video synthesis with a significantly smaller memory footprint and faster decoding than traditional video diffusion models.

Introduction

Current state-of-the-art video generation typically relies on Video Latent Diffusion Models (LDMs), which encode videos as sequences of individual frame-wise feature maps. This approach suffers from several drawbacks: high redundancy by ignoring inter-frame coherence, extreme computational costs due to heavy cross-frame attention layers, and slow decoding times from large Variational Autoencoders (VAEs).

NeRV-Diffusion addresses these issues by leveraging Implicit Neural Representations (INRs)—networks where weights themselves store visual information. While INRs are efficient, their weights are typically not "well-behaved" for standard diffusion models. NeRV-Diffusion introduces a method to make these neural weights "diffusable," enabling efficient video generation in weight space.

Methodology

The NeRV-Diffusion pipeline consists of three primary components designed to bridge the gap between INRs and diffusion models:

  1. The NeRV Autoencoder (Tokenizer): A hypernetwork-based encoder compresses raw video into "latent weight tokens." A KL Bottleneck forces these tokens toward a Gaussian distribution, while Channel-wise Parameterization allows tokens to directly define the convolution kernels of the decoder.
  2. The Implicit Diffusion Transformer (Implicit DiT): This generative engine operates entirely in weight space, iteratively denoising random noise to produce final INR weights.
  3. The Generative NeRV Decoder: An efficient convolutional architecture that uses spatiotemporal embeddings to decode the generated weights into a full video.
Overview of the NeRV-Diffusion framework, illustrating the transition from video to neural weights and the diffusion process in weight space.

Because the video is represented as a single set of weights, temporal consistency is inherently "baked into" the representation, eliminating the need for expensive temporal attention layers during denoising.

Key Insights

  • Weight Synthesis as Meta-Learning: Video synthesis is treated as generating a function (the weights) rather than a grid of pixels.
  • Holistic Compression: The INR captures the "keyframe-residue" relationship naturally, avoiding the need for explicit motion vectors or optical flow.
  • Latent Space Smoothness: The KL bottleneck enables a smooth latent space, allowing for seamless interpolation between different video weights.
  • Efficiency: The model achieves high performance with a fraction of the parameters used by models like MAGVIT; for instance, the "Small" version uses only 3.5M parameters for the detokenizer.

Experimental Results

NeRV-Diffusion was evaluated on standard benchmarks including UCF-101, Kinetics-600, and TaiChi-HD.

Performance comparison showing gFVD vs. NeRV Decoder size. NeRV-Diffusion variants (S, B, L) demonstrate superior efficiency and quality compared to existing models.

Key findings include:

  • State-of-the-Art Quality: NeRV-Diffusion-L achieved a gFVD of 97 on UCF-101, outperforming previous INR-based models like DIGAN and remaining competitive with much larger non-implicit models.
  • Compact Representation: The framework uses only 128 tokens to represent a video, compared to 1024+ tokens in models like TATS or MAGVIT.
  • Temporal Flexibility: The continuous time coordinate input allows for 8x interpolation (e.g., turning a 16-frame video into 128 frames) without retraining.

Conclusion and Future Work

NeRV-Diffusion demonstrates that generating neural weights is a viable and efficient alternative to pixel-space video synthesis. While current scaling is limited to $128 \times 128$ resolution, future work aims to address high-definition scaling and complex text-to-video conditioning. This "weight-generation" paradigm also holds promise for 3D scene synthesis applications.

Summary updated

Abstract

We present NeRV-Diffusion, an implicit latent video diffusion model that synthesizes videos via generating neural network weights. The generated weights can be rearranged as the parameters of a convolutional neural network, which forms an implicit neural representation (INR), and decodes into videos with frame indices as the input. Our framework consists of two stages: 1) A hypernetworkbased tokenizer that encodes raw videos from pixel space to neural parameter space, where the bottleneck latent serves as INR weights to decode. 2) An implicit diffusion transformer that denoises on the latent INR weights. In contrast to traditional video tokenizers that encode videos into frame-wise feature maps, NeRV-Diffusion compresses and generates a video holistically as a unified neural network. This enables efficient and high-quality video synthesis via obviating temporal cross-frame attentions in the denoiser and decoding video latent with dedicated decoders. To achieve Gaussian-distributed INR weights with high expressiveness, we reuse the bottleneck latent across all NeRV layers, as well as reform its weight assignment, upsampling connection and input coordinates. We also introduce SNR-adaptive loss weighting and scheduled sampling for effective training of the implicit diffusion model. NeRV-Diffusion reaches superior video generation quality over previous INR-based models and comparable performance to most recent state-of-the-art non-implicit models on real-world video benchmarks including UCF-101 and Kinetics-600. It also brings a smooth INR weight space that facilitates seamless interpolations between frames or videos.
BibTeX
@misc{ren2025nerv,
  author = {Yixuan Ren and Hanyu Wang and Hao Chen and Bo He and Abhinav Shrivastava},
  title = {{NeRV-Diffusion: Diffuse Implicit Neural Representations for Video Synthesis}},
  year = {2025},
  eprint = {2509.24353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24353},
}
Figure