TL;DR
This paper introduces a model-based video compression technique for the AV1 codec that uses a Convolutional Neural Network (CNN) to identify "perceptually insignificant" texture regions. By synthesizing these textures using global motion parameters rather than encoding every pixel, the authors achieve significant bitrate reductions—up to 16%—while maintaining high visual quality.
What Problem Does This Paper Address?
Modern video compression standards like HEVC and AV1 primarily rely on minimizing Mean Squared Error (MSE) to achieve efficiency. However, MSE doesn't always align with human perception. Certain areas of a video, such as grass, water, or gravel, contain dense textures where the human eye doesn't track specific pixel-level details.
Traditional codecs spend a significant portion of their bit budget trying to perfectly reconstruct these textures. The authors argue that this is inefficient. If we can identify these "detail-irrelevant" regions, we can use a simpler model to synthesize them at the decoder, saving a massive amount of data without the viewer noticing a drop in quality.
Methods
The authors propose a hybrid pipeline that integrates a deep-learning-based texture detector into the AV1 encoding process.
-
Texture Detection (CNN): The system divides each frame into 16x16 blocks. A VGG-inspired CNN classifies each block as either "texture" or "non-texture." The architecture (shown in Figure 2) uses convolutional layers, batch normalization, and max-pooling, followed by three fully connected layers to produce a classification probability.
CNN Architecture for Texture Block Classification -
Texture Synthesis: For blocks identified as texture, the encoder skips the standard inter-frame prediction. Instead, it estimates a single set of global motion parameters for the entire texture region.
-
Encoding Strategy: As illustrated in Figure 4, if a block is deemed a texture block, the encoder chooses the
ZEROMVmode and sets the residual (the difference between the original and predicted pixels) to zero. This effectively tells the decoder to just "warp" the texture from a reference frame using the provided motion parameters.Encoding Operation Flow
Datasets
- Training: The authors used the Salzburg Texture Image Database (STex) for "pure" texture samples and the Places365 dataset for general, non-texture scenes. To prepare the data, they cropped and scaled images to match the 16x16 block size used in the detector.
- Evaluation: The method was tested on standard video sequences, specifically "Flower garden" and "Football," which contain prominent textured areas like grass and trees.
Key Insights and Ideas
- Perceptual Redundancy: The core "aha" moment is leveraging the fact that the human visual system is satisfied by statistical consistency in textures rather than pixel-perfect accuracy.
- Zeroing the Residual: By setting the residual to zero for texture blocks, the authors eliminate the most data-intensive part of the bitstream for those regions.
- CNN vs. Hand-tuned Features: Previous model-based approaches required manual parameter tuning for different video types. Moving to a CNN-based segmentation (Figure 7) allows for a more automated and robust inference process that generalizes better across different content.
Key Results
The preliminary results demonstrate a clear trade-off advantage:
- Flower Garden: Achieved a 16.49% reduction in BD-RATE (bitrate) and a 1.25 dB improvement in BD-PSNR (calculated on non-texture regions).
- Football: Achieved a 5.68% reduction in BD-RATE.
- Visual Quality: Qualitative results (Figure 5) show that the reconstructed frames remain visually pleasing despite the aggressive bit savings.
Limitations and Future Work
While promising, the method faces challenges with complex motion. In the "Football" and "Table Tennis" sequences, fast or non-uniform motion can lead to artifacts at the boundaries between texture and non-texture regions (see Figure 6).
Future directions include: * Advanced Motion Models: Moving beyond simple global motion to handle more complex movements. * Detector Refinement: Improving the CNN to handle blocks containing multiple texture types, which currently confuse the classifier. * Post-processing: Implementing techniques like connected component analysis to clean up the segmentation masks and remove small, isolated texture regions that might cause flickering.
Summary updated