Multi-Reference Video Coding Using Stillness Detection

Di Chen, Zoe Liu, Yaowu Xu, Fengqing Zhu, Edward DelparXiv 2018 · arXiv

TL;DR

This paper introduces an adaptive approach to video coding in the AOM/AV1 codec by automatically detecting "stillness" in video segments. By switching from a complex multilayer coding structure to a simpler one-layer structure when a scene is static, the authors achieve significant compression gains (up to ~1% BDRATE reduction) without sacrificing quality.

What Problem Does This Paper Address?

Modern video codecs like AV1 organize video frames into "Golden-Frame" (GF) groups. Traditionally, these groups use a fixed, hierarchical multilayer coding structure. This structure uses multiple reference frames (looking both forward and backward in time) to predict the content of the current frame, which is highly effective for scenes with complex motion.

However, the authors observed a "diminishing returns" problem: for very still or static scenes, this multilayer complexity actually hurts performance. The overhead of managing multiple reference frames and the specific way bits are allocated across layers becomes inefficient when the frames are nearly identical. The challenge is to automatically identify these "still" segments during encoding and switch to a more efficient, simpler coding structure.

Methods

The researchers proposed a content-adaptive system that chooses between two coding structures:

  1. Multilayer Structure (Standard): Uses a complex hierarchy of reference frames (e.g., LAST, BWDREF, ALTREF) to handle motion.
  2. One-Layer Structure (Proposed for Stillness): A simplified approach that reduces the number of backward reference frames, minimizing overhead for static content.
Comparison of multilayer vs. one-layer coding structures.

To decide which structure to use, the encoder performs a fast "first pass" to extract three key metrics for each GF group:

  • Zero Motion Accumulator: The minimum percentage of blocks in a frame that require zero motion vectors. High values indicate a static scene.
  • Average Pixel Error: The mean Sum of Squared Errors (SSE) per pixel. Low error suggests the frames are very similar to their predecessors.
  • Average Error Standard Deviation: Measures the consistency of prediction error across the frame.

The system uses a simple decision workflow: if a GF group meets specific thresholds for all three metrics, it is classified as "still" and encoded using the one-layer structure.

The decision workflow for stillness detection.

Datasets

The authors evaluated their method using two standard Google video test sets:

  • Lowres Set: 40 video clips at CIF resolution (352x288).
  • Midres Set: 30 video clips at 360p and 480p resolutions.

Within these sets, specific clips like pamplet_cif, bowing_cif, and snow_mnt were identified as containing significant still segments for detailed analysis.

Key Insights and Ideas

  • Complexity Isn't Always Better: In video compression, adding more reference frames usually improves prediction. However, this paper highlights that the signaling overhead (telling the decoder which reference to use) and unbalanced bit allocation can outweigh the prediction benefits in static scenes.
  • Thresholding for Robustness: The authors found that using a single metric wasn't enough to reliably detect stillness. By combining three metrics (motion, absolute error, and error distribution), they created a robust classifier that avoided misclassifying "still-like" scenes (e.g., very slow motion) which still benefit from multilayer coding.
  • Bit Allocation Matters: One reason the multilayer structure fails for still scenes is its "unbalanced bit allocation scheme." In a static scene, frames are equally important, but multilayer structures tend to prioritize certain layers, which is counterproductive when temporal correlation is near-perfect.

Key Results

The adaptive approach demonstrated consistent coding gains:

  • BDRATE Reduction: For clips identified with stillness, the method achieved a BDRATE reduction of approximately 1% (lower is better, indicating more efficient compression for the same quality).
  • Specific Wins: The pamplet_cif clip saw a PSNR-based BDRATE reduction of 1.395%, while snow_mnt saw an SSIM-based reduction of 1.235%.
  • Accuracy: The automatic detector resulted in zero misclassifications across the test sets, ensuring that non-still videos were not negatively impacted by the simplified coding structure.

Limitations and Future Work

While effective, the current approach only switches between two specific structures based on stillness. The authors note that other video characteristics—such as rapid zooming or extremely high motion—might also benefit from their own specialized coding structures.

Future research could expand this adaptive logic to a wider variety of content types, potentially creating a library of coding structures that the encoder can select from dynamically based on a broader suite of real-time content analysis metrics.

Summary updated

Abstract

Encoders of AOM/AV1 codec consider an input video sequence as succession of frames grouped in Golden-Frame (GF) groups. The coding structure of a GF group is fixed with a given GF group size. In the current AOM/AV1 encoder, video frames are coded using a hierarchical, multilayer coding structure within one GF group. It has been observed that the use of multilayer coding structure may result in worse coding performance if the GF group presents consistent stillness across its frames. This paper proposes a new approach that adaptively designs the Golden-Frame (GF) group coding structure through the use of stillness detection. Our new approach hence develops an automatic stillness detection scheme using three metrics extracted from each GF group. It then differentiates those GF groups of stillness from other non- still GF groups and uses different GF coding structures accordingly. Experimental result demonstrates a consistent coding gain using the new approach.
BibTeX
@misc{chen2018multi,
  author = {Di Chen and Zoe Liu and Yaowu Xu and Fengqing Zhu and Edward Delp},
  title = {{Multi-Reference Video Coding Using Stillness Detection}},
  year = {2018},
  eprint = {1803.04061},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.04061},
}

Loading related content...

Figure