Video processing and compression technologies

Chuanmin Jia, Haichuan Ma, Wenhan Yang, Wenqi Ren, Jinshan Pan, Dong Liu, Jiaying Liu, Siwei MaJournal of Image and Graphics 2021 · DOI

Abstract

视频处理与压缩是多媒体计算与通信领域的核心主题之一,是连接视频采集传输和视觉分析理解的关键桥梁,也是诸多视频应用的基础。当前“5G+超高清+AI”正在引发多媒体计算与通信领域的新一轮重大技术革新,视频处理与压缩技术正在发生深刻变革,亟需建立视频大数据高效紧凑表示理论和方法。本文从数字信号处理基础理论出发,分析了当前视频处理与压缩领域的热点问题和研究内容,包括基于统计先验模型的视频数据表示模型及处理方法、融合深度网络模型的视频处理技术、视频压缩技术以及视频压缩标准进展等领域。详细描述了视频超分辨率、视频重建与恢复、视频压缩技术等领域面临的前沿动态、发展趋势、技术瓶颈和标准化进程等内容,对国际国内研究内容和发展现状进行了综合对比与分析,并展望了视频处理与压缩技术的发展与演进方向。

Introduction

Annual Report on the Development of Image and Graphics Source, p. 1

Annual Report on the Development of Image and Graphics

We are living in an era of unprecedented video data explosion. With the advent of 5G and Ultra-High-Definition (UHD) content, traditional video processing and compression techniques are reaching their performance ceilings. The core challenge is twofold:

  1. Quality Enhancement: Effectively restoring and upscaling video (super-resolution, deblurring, deraining) to meet the demands of high-end displays.
  2. Efficiency: Compressing massive amounts of data for transmission without losing the perceptual quality required for both human viewers and machine learning analysis.

This review provides a roadmap of how the field is moving away from hand-crafted, modular pipelines toward integrated, data-driven neural networks.

Technological Landscape

The paper categorizes the technological landscape into two main pillars: processing/restoration and compression.

1. Video Processing & Restoration

  • Super-Resolution (SR): The field contrasts "alignment-based" methods (using motion compensation or optical flow to sync frames) with "non-alignment" methods (using 3D convolutions or recurrent networks like BRCN).
  • Restoration: Techniques for removing noise like rain or snow are moving from time-domain/frequency-domain statistical models to deep CNN architectures that can handle complex, dynamic backgrounds.

2. Video Compression

  • Traditional Hybrid Coding: The evolution of the block-based framework (partitioning, prediction, transform, quantization) continues with flexible partitioning in VVC (H.266) and AVS3.
  • Deep Learning Enhanced Coding: This involves replacing specific modules of traditional codecs with neural networks, such as using CNNs for in-loop filtering or intra-prediction.
  • End-to-End Neural Video Compression (NVC): A radical approach where the entire codec is a single, differentiable neural network (often based on autoencoders or RNNs), optimized directly for rate-distortion performance.

Key Insights and Results

  • The Convergence of Processing and Coding: Historically, restoration and compression were separate tasks. The future lies in "joint optimization," where the coding process is aware of downstream visual tasks.
  • VVC vs. AVS3: There is significant parity between international standards (VVC) and domestic Chinese standards (AVS3). Both represent the state-of-the-art, achieving massive bitrate savings over the previous generation (HEVC).
  • Compression Efficiency: The latest standard, VVC (H.266), achieves approximately a 36.6% bitrate reduction over HEVC/H.265 while maintaining the same objective quality. Rate-Distortion analysis shows that VVC and AVS3 currently lead the industry in the UHD category.
  • Machine-Centric Coding: Compression is no longer just for humans. As more video is consumed by AI (e.g., surveillance, autonomous driving), codecs must evolve to preserve features essential for machine analysis rather than just perceptual aesthetics.

Limitations and Future Work

  • Computational Complexity: The primary hurdle for deep learning-based video tech is the massive computational overhead. Real-time encoding/decoding using complex neural networks remains a challenge for consumer hardware.
  • Generalization: Many AI models perform exceptionally well on specific datasets but struggle with the "long tail" of diverse, real-world video content.
  • Future Directions: Development is pointing toward more efficient "lightweight" models, the standardization of neural-based coding tools, and the deeper integration of "low-level" processing with "high-level" semantic understanding.
Journal of Image and Graphics Logo

Summary updated

Abstract

视频处理与压缩是多媒体计算与通信领域的核心主题之一,是连接视频采集传输和视觉分析理解的关键桥梁,也是诸多视频应用的基础。当前“5G+超高清+AI”正在引发多媒体计算与通信领域的新一轮重大技术革新,视频处理与压缩技术正在发生深刻变革,亟需建立视频大数据高效紧凑表示理论和方法。为此,学术研究机构和工业界对视频大数据的视觉表示机理、视觉信息紧凑表达、视频信号重建与恢复、高层与低层视觉融合处理方法及相应硬件技术等前沿领域进行了广泛深入研究。本文从数字信号处理基础理论出发,分析了当前视频处理与压缩领域的热点问题和研究内容,包括基于统计先验模型的视频数据表示模型及处理方法、融合深度网络模型的视频处理技术、视频压缩技术以及视频压缩标准进展等领域。详细描述了视频超分辨率、视频重建与恢复、视频压缩技术等领域面临的前沿动态、发展趋势、技术瓶颈和标准化进程等内容,对国际国内研究内容和发展现状进行了综合对比与分析,并展望了视频处理与压缩技术的发展与演进方向。更高质量视觉效果和高效率视觉表达之间将不再是单独研究的个体,融合类脑视觉系统及编码机理的视频处理与压缩技术将是未来研究的重要领域之一。
BibTeX
@article{jia2021video,
  author = {Chuanmin Jia and Haichuan Ma and Wenhan Yang and Wenqi Ren and Jinshan Pan and Dong Liu and Jiaying Liu and Siwei Ma},
  title = {{Video processing and compression technologies}},
  journal = {Journal of Image and Graphics},
  year = {2021},
  doi = {10.11834/jig.200861},
}
Figure