TL;DR
This paper introduces an adaptive approach to video coding in the AOM/AV1 codec by automatically detecting "stillness" in video segments. By switching from a complex multilayer coding structure to a simpler one-layer structure when a scene is static, the authors achieve significant compression gains (up to ~1% BDRATE reduction) without sacrificing quality.
What Problem Does This Paper Address?
Modern video codecs like AV1 organize video frames into "Golden-Frame" (GF) groups. Traditionally, these groups use a fixed, hierarchical multilayer coding structure. This structure uses multiple reference frames (looking both forward and backward in time) to predict the content of the current frame, which is highly effective for scenes with complex motion.
However, the authors observed a "diminishing returns" problem: for very still or static scenes, this multilayer complexity actually hurts performance. The overhead of managing multiple reference frames and the specific way bits are allocated across layers becomes inefficient when the frames are nearly identical. The challenge is to automatically identify these "still" segments during encoding and switch to a more efficient, simpler coding structure.
Methods
The researchers proposed a content-adaptive system that chooses between two coding structures:
- Multilayer Structure (Standard): Uses a complex hierarchy of reference frames (e.g.,
LAST,BWDREF,ALTREF) to handle motion. - One-Layer Structure (Proposed for Stillness): A simplified approach that reduces the number of backward reference frames, minimizing overhead for static content.
To decide which structure to use, the encoder performs a fast "first pass" to extract three key metrics for each GF group:
- Zero Motion Accumulator: The minimum percentage of blocks in a frame that require zero motion vectors. High values indicate a static scene.
- Average Pixel Error: The mean Sum of Squared Errors (SSE) per pixel. Low error suggests the frames are very similar to their predecessors.
- Average Error Standard Deviation: Measures the consistency of prediction error across the frame.
The system uses a simple decision workflow: if a GF group meets specific thresholds for all three metrics, it is classified as "still" and encoded using the one-layer structure.
Datasets
The authors evaluated their method using two standard Google video test sets:
- Lowres Set: 40 video clips at CIF resolution (352x288).
- Midres Set: 30 video clips at 360p and 480p resolutions.
Within these sets, specific clips like pamplet_cif, bowing_cif, and snow_mnt were identified as containing significant still segments for detailed analysis.
Key Insights and Ideas
- Complexity Isn't Always Better: In video compression, adding more reference frames usually improves prediction. However, this paper highlights that the signaling overhead (telling the decoder which reference to use) and unbalanced bit allocation can outweigh the prediction benefits in static scenes.
- Thresholding for Robustness: The authors found that using a single metric wasn't enough to reliably detect stillness. By combining three metrics (motion, absolute error, and error distribution), they created a robust classifier that avoided misclassifying "still-like" scenes (e.g., very slow motion) which still benefit from multilayer coding.
- Bit Allocation Matters: One reason the multilayer structure fails for still scenes is its "unbalanced bit allocation scheme." In a static scene, frames are equally important, but multilayer structures tend to prioritize certain layers, which is counterproductive when temporal correlation is near-perfect.
Key Results
The adaptive approach demonstrated consistent coding gains:
- BDRATE Reduction: For clips identified with stillness, the method achieved a BDRATE reduction of approximately 1% (lower is better, indicating more efficient compression for the same quality).
- Specific Wins: The
pamplet_cifclip saw a PSNR-based BDRATE reduction of 1.395%, whilesnow_mntsaw an SSIM-based reduction of 1.235%. - Accuracy: The automatic detector resulted in zero misclassifications across the test sets, ensuring that non-still videos were not negatively impacted by the simplified coding structure.
Limitations and Future Work
While effective, the current approach only switches between two specific structures based on stillness. The authors note that other video characteristics—such as rapid zooming or extremely high motion—might also benefit from their own specialized coding structures.
Future research could expand this adaptive logic to a wider variety of content types, potentially creating a library of coding structures that the encoder can select from dynamically based on a broader suite of real-time content analysis metrics.
Summary updated