Neural Residual Flow Fields for Efficient Video Representations

Daniel Rho, Junwoo Cho, Jong Hwan Ko, Eunbyung Park

Introduction

Implicit Neural Representations (INRs) have emerged as a powerful paradigm for representing continuous signals by mapping coordinates to values via neural networks. However, applying naive INRs to video data is often inefficient. Standard models attempt to memorize the color of every pixel at every time step, failing to exploit the inherent temporal redundancy where most frames are simply shifted versions of their predecessors. This leads to high parameter counts and blurry reconstructions.

Neural Residual Flow Fields (NRFF)

To address these inefficiencies, we propose Neural Residual Flow Fields (NRFF). Instead of directly predicting RGB values, the network is trained to predict optical flow and residuals.

The NRFF pipeline: A central key frame is warped using predicted flow fields and corrected with residuals to reconstruct neighboring frames.

Core Components

  1. Flow-Based Representation: The network outputs motion vectors (optical flow) that describe how pixels from a reference "key frame" should be moved to form the target frame.
  2. Warping and Residual Correction: A high-quality key frame is warped using the network's flow. Because warping cannot account for new information (dis-occlusions), a residual map is generated to fill gaps and refine details.
  3. Multi-Reference Blending: For complex sequences, the model utilizes multiple reference frames. It learns pixel-wise weights to blend information from past and future frames effectively.
Detailed view of the multi-reference blending mechanism, showing the combination of warped references, learned weights, and residuals to generate the final frame.
  1. Network Splitting: By using separate subnetworks for motion (flow) and appearance changes (residuals), the model prevents signal interference, leading to better parameter efficiency.

Experimental Results

NRFF was evaluated on the MPI Sintel and UVG datasets. By focusing on learning smooth motion fields rather than complex textures, the model "borrows" high-frequency details from reference frames, resulting in significantly sharper visual outputs.

Quantitatively, NRFF achieves a substantial leap over baseline coordinate-based models like SIREN. On the MPI Sintel dataset, PSNR improved from 30 dB to 37 dB, and SSIM increased from 0.85 to 0.95 at comparable bitrates. Qualitatively, NRFF preserves fine details—such as water droplets and textures—that are typically lost in standard neural representations.

Conclusion and Future Work

NRFF demonstrates that incorporating motion priors into neural fields significantly enhances video representation efficiency. While training time remains a challenge compared to traditional codecs, the representation's ability to outperform recent neural methods like NeRV and IPF suggests a promising path forward. Future work will focus on integrating advanced compression techniques like quantization and pruning to further optimize the pipeline for real-world deployment.

Summary updated

BibTeX
@inproceedings{rho2022neural,
  author = {Daniel Rho and Junwoo Cho and Jong Hwan Ko and Eunbyung Park},
  title = {{Neural Residual Flow Fields for Efficient Video Representations}},
  booktitle = {ACCV},
  year = {2022},
  doi = {10.1007/978-3-031-26284-5\_28},
}
Figure