PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos

Qi Zhao, M. Salman Asif, Zhan Ma

Abstract

Implicit Neural Representations (INRs) for video, commonly referred to as NeRV (Neural Representation for Videos), have revolutionized video compression and synthesis by encoding video data into the weights of a neural network. However, existing NeRV frameworks often suffer from spatial inconsistency and perceptual blurring due to their reliance on local upsampling operators. We present PNeRV (Pyramidal Neural Representation for Videos), a novel architecture that utilizes a multi-scale pyramidal structure to address these limitations. By introducing the Kronecker Fully-connected (KFc) layer for global-wise upscaling and the Benign Selective Memory (BSM) module for adaptive feature fusion, PNeRV significantly enhances spatial consistency. Experimental results demonstrate that PNeRV outperforms state-of-the-art models, achieving a 634% improvement in perceptual quality (FVD) on challenging datasets like DAVIS.

1. Introduction

Implicit Neural Representations (INRs) have emerged as a compelling alternative to traditional grid-based video storage. By treating a video as a continuous function mapping a time index to a frame, NeRV models allow for efficient compression and flexible synthesis. Despite their potential, current NeRV architectures frequently produce "shimmering" artifacts and blurred high-frequency details.

This "blurry problem" stems from a lack of global context. Most models utilize local upsampling operators (e.g., PixelShuffle or standard convolutions) that lack a holistic view of the frame. Consequently, the model struggles with semantic uncertainty—failing to maintain consistency across distant pixels belonging to the same object. PNeRV addresses this by shifting from a serial mainstream architecture to a hierarchical pyramidal design that prioritizes global coordination.

2. Methodology

2.1 Kronecker Fully-connected (KFc) Layer

To overcome the limitations of local operators, PNeRV introduces the Kronecker Fully-connected (KFc) layer. Standard fully-connected layers for high-resolution frames are computationally prohibitive. The KFc layer utilizes Kronecker decomposition to perform global-wise upscaling with extreme efficiency. By calculating correlations across every position in a feature map using only 0.7% of the parameters required by standard PixelShuffle, the KFc layer provides the global receptive field necessary to maintain structural consistency across the entire frame.

2.2 Benign Selective Memory (BSM)

Feature fusion in hierarchical models often leads to high-level semantic information being overwhelmed by low-level pixel noise. PNeRV introduces Benign Selective Memory (BSM), a gated mechanism inspired by Recurrent Neural Networks. BSM treats high-level "Knowledge" (from pyramidal shortcuts) and fine-grained "Memory" (from the mainstream path) as distinct streams. An adaptive gating process regulates the flow of information, ensuring that semantic clarity is preserved during the fusion process.

2.3 Universal Approximation Theory (UAT) Analysis

The design of PNeRV is supported by a theoretical analysis using Universal Approximation Theory. The authors demonstrate that the inclusion of pyramidal shortcuts reduces the minimal width required for a neural network to approximate complex video functions. This provides a mathematical justification for the efficiency of the PNeRV architecture, proving that shortcuts are essential for maintaining high resolution within a manageable parameter budget.

3. Experimental Evaluation

3.1 Datasets and Metrics

The model was evaluated on two benchmark datasets:

  • UVG (Ultra Video Group): Comprising 7 high-quality videos at 120 FPS, used to test motion smoothness and reconstruction fidelity.
  • DAVIS: A subset of 9 videos featuring complex spatial details, camera rotations, and extreme dynamics (e.g., "Parkour").

Performance was measured using Peak Signal-to-Noise Ratio (PSNR) for reconstruction accuracy and Frechet Video Distance (FVD) for perceptual quality.

3.2 Quantitative Results

PNeRV establishes a new state-of-the-art for video regression:

  • PSNR Performance: On the UVG dataset, PNeRV-L achieved a +4.49 dB gain over vanilla NeRV and a +0.42 dB improvement over DiffNeRV.
  • Perceptual Quality: The most significant leap was observed in FVD scores. PNeRV achieved a 231% improvement on UVG and a staggering 634% increase on the DAVIS dataset compared to the vanilla NeRV baseline.

4. Discussion and Conclusion

4.1 Insights

The success of PNeRV highlights that video reconstruction is fundamentally a global coordination task rather than a series of local upscaling steps. By replacing local operators with the KFc layer, the model can maintain sharp edges and consistent textures across large objects. Furthermore, the disentangled fusion provided by the BSM module prevents the "washing out" of semantic details, which is critical for high-fidelity synthesis.

4.2 Limitations and Future Work

Despite its performance, PNeRV's hierarchical structure increases computational complexity, resulting in higher FLOPs during the decoding phase. Additionally, while spatial consistency is greatly improved, the model still faces challenges with extreme temporal discontinuities, such as rapid camera cuts. Future research will focus on optimizing the architecture for faster inference and improving robustness against severe temporal jitter.

Summary updated

Abstract

The primary focus of Neural Representation for Videos (NeRV) is to effectively model its spatiotemporal consistency. However, current NeRV systems often face a significant issue of spatial inconsistency, leading to decreased perceptual quality. To address this issue, we introduce the Pyramidal Neural Representation for Videos (PNeRV), which is built on a multi-scale information connection and comprises a lightweight rescaling operator, Kronecker Fully-connected layer (KFc), and a Benign Selective Memory (BSM) mechanism. The KFc, inspired by the tensor decomposition of the vanilla Fully-connected layer, facilitates low-cost rescaling and global correlation modeling. BSM merges high-level features with granular ones adaptively. Furthermore, we provide an analysis based on the Universal Approximation Theory of the NeRV system and validate the effectiveness of the proposed PNeRV.We conducted comprehensive experiments to demonstrate that PNeRV surpasses the performance of contemporary NeRV models, achieving the best results in video regression on UVG and DAVIS under various metrics (PSNR, SSIM, LPIPS, and FVD). Compared to vanilla NeRV, PNeRV achieves a +4.49 dB gain in PSNR and a 231% increase in FVD on UVG, along with a +3.28 dB PSNR and 634% FVD increase on DAVIS.
BibTeX
@inproceedings{zhao2024pnerv,
  author = {Qi Zhao and M. Salman Asif and Zhan Ma},
  title = {{PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure