Compressing Flow Fields with Edge-Aware Homogeneous Diffusion Inpainting

Ferdinand Jost, Pascal Peter, Joachim Weickert

Abstract

In spite of the fact that efficient compression methods for dense two-dimensional flow fields would be very useful for modern video codecs, hardly any research has been performed in this area so far. Our paper addresses this problem by proposing the first lossy diffusion-based codec for this purpose. It keeps only a few flow vectors on a coarse grid. Additionally stored edge locations ensure the accurate representation of discontinuities. In the decoding step, the missing information is recovered by homogeneous diffusion inpainting that incorporates the stored edges as reflecting boundary conditions. In spite of the simple nature of this codec, our experiments show that it achieves remarkable quality for compression ratios up to 800:1.

Introduction

In video compression and computer vision, dense flow fields (optical flow) describe how pixels move between frames. While modern video codecs like HEVC use simple block-based motion vectors to save space, these often create "blocky" artifacts. Dense flow fields—where every single pixel has a motion vector—provide much better predictions but are notoriously difficult to compress efficiently due to their high data density.

Despite their importance, there are few dedicated codecs for dense flow fields. Most research either employs general-purpose image compressors or attempts to integrate compression directly into the flow estimation process. This paper fills that gap by treating flow field compression as an inpainting problem: by saving only a tiny fraction of the data, the missing information can be mathematically reconstructed with high accuracy.

Methodology

The authors propose a codec based on Edge-Aware Homogeneous Diffusion Inpainting. The process follows a multi-stage pipeline:

  1. Edge Detection and Encoding: Using the Marr-Hildreth operator combined with hysteresis thresholding, the encoder identifies critical boundaries where motion changes abruptly. To ensure memory efficiency, these edges are encoded using chain codes, identifying "T-junctions" as starting points to trace contours.
  2. Inpainting Mask: The encoder saves actual flow vectors only on a coarse, regular grid (e.g., every $d$-th pixel). For areas isolated by edges that do not contain a grid point, a single average flow value is stored.
  3. Quantization and Entropy Coding: The remaining data is quantized and compressed using lpaq2, a powerful context-mixing coder.
  4. Decoding via Diffusion: To reconstruct the field, the decoder solves the Laplace equation. It treats the stored grid points as fixed values and "diffuses" the information into the empty spaces. Crucially, the stored edges act as reflecting boundary conditions, preventing motion data from "leaking" across object boundaries and ensuring sharp edges remain crisp.

A key insight of this work is that while natural images often require complex anisotropic diffusion due to texture, flow fields are generally piecewise smooth. This makes them ideal candidates for homogeneous diffusion inpainting, provided an accurate edge map is available to act as a barrier.

Experimental Results

The proposed method was evaluated using the MPI Sintel Flow Dataset, a standard benchmark derived from an animated short film that provides ground truth flow fields with realistic complexities like motion blur and non-rigid transformations.

Visualization of a dense flow field from the MPI Sintel dataset.

The "Edge-aware" approach was compared against JPEG2000 and BPG (the still-image mode of the HEVC/H.265 codec). Quantitative results demonstrate that the proposed codec consistently outperforms both benchmarks in terms of Peak Signal-to-Noise Ratio (PSNR) across all tested compression ratios, from 10:1 to 800:1.

PSNR performance comparison across various compression ratios for two sequences.

For the "alley" sequence at a 400:1 compression ratio, the proposed method achieved a PSNR of 46.10 dB, compared to BPG’s 41.73 dB and JPEG2000’s 36.47 dB. Visually, at high compression ratios, JPEG2000 suffered from heavy ringing artifacts and BPG blurred edges significantly, while the proposed method maintained crisp boundaries and clean smooth regions.

Discussion and Future Work

Currently, the codec operates on an intra-frame basis, treating every frame independently without exploiting temporal redundancy. Furthermore, solving the Laplace equation via the conjugate gradient method is more computationally intensive than the inverse transforms used in standard codecs. Future work will focus on integrating temporal prediction and optimizing the implementation for real-time decoding to facilitate adoption in full video compression pipelines.

Summary updated

Abstract

In spite of the fact that efficient compression methods for dense two-dimensional flow fields would be very useful for modern video codecs, hardly any research has been performed in this area so far. Our paper addresses this problem by proposing the first lossy diffusion-based codec for this purpose. It keeps only a few flow vectors on a coarse grid. Additionally stored edge locations ensure the accurate representation of discontinuities. In the decoding step, the missing information is recovered by homogeneous diffusion inpainting that incorporates the stored edges as reflecting boundary conditions. In spite of the simple nature of this codec, our experiments show that it achieves remarkable quality for compression ratios up to 800:1.
BibTeX
@inproceedings{jost2020compressing,
  author = {Ferdinand Jost and Pascal Peter and Joachim Weickert},
  title = {{Compressing Flow Fields with Edge-Aware Homogeneous Diffusion Inpainting}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=q3JlbZecU3u},
}
Figure