An Emerging Coding Paradigm Vcm: A Scalable Coding Approach Beyond Feature And Signal

Sifeng Xia, Kunchangtai Liang, Wenhan Yang, Ling-Yu Duan, Jiaying Liu

Abstract

Video Coding for Machine (VCM) is an emerging paradigm that bridges the gap between human-centric video compression and AI-centric feature extraction. By using sparse motion patterns as a "bridge," this paper proposes a scalable framework that achieves superior action recognition at 1/3 the bitrate of HEVC while providing higher-quality video reconstruction for human viewers.

Introduction

Traditional video codecs, such as H.264 (AVC) and H.265 (HEVC), are primarily optimized for human perception, focusing on minimizing signal-level distortion. However, in modern applications like smart cities and big data analytics, the majority of video content is consumed by machines for tasks such as action recognition and event detection.

Current compression strategies face a significant trade-off: human-centric compression often introduces artifacts that hinder AI performance, while machine-centric feature extraction typically discards the signal data necessary for human verification. This paper introduces a scalable VCM approach that satisfies both requirements by layering machine-vision features with signal-reconstruction data.

Proposed Methodology

The proposed framework integrates feature and video compression through two core components: the Sparse Point Prediction Network (SPPN) and the Motion Guided Generation Network (MGGN).

The proposed joint feature and video compression framework, illustrating the scalable path from sparse points to full video reconstruction.

Sparse Point Prediction Network (SPPN)

Instead of transmitting dense pixel data, the encoder identifies a "sparse motion pattern." Guided by human skeleton data, the SPPN extracts key points representing object movement. Each point is represented by its coordinates and a covariance matrix, resulting in an extremely compact representation (6 float numbers per point).

Motion Guided Generation Network (MGGN)

At the decoder, the MGGN—a conditional deep generative model—uses the reconstructed keyframes and sparse motion patterns to synthesize intermediate frames. It estimates motion flow and deforms keyframe features to match the target pose. This generative approach differs from traditional predictive coding by "hallucinating" realistic pixels based on learned patterns.

Experimental Evaluation

The framework was evaluated using the PKU-MMD dataset, which provides 3D human action data. The model was trained on 3,317 clips and tested on 227 clips, using ground-truth skeleton joints to supervise the SPPN.

Machine Vision Performance

The VCM approach significantly outperformed HEVC in action recognition tasks. Specifically, the proposed method achieved 74.6% accuracy at 5.2 Kbps, whereas HEVC achieved only 65.2% accuracy at a much higher bitrate of 16.2 Kbps. This represents a 9.4% accuracy gain at approximately one-third the bitrate.

Human Vision Performance

For video reconstruction, the VCM method achieved an SSIM of 0.9071 at ~33 Kbps, surpassing HEVC’s 0.9008. Qualitatively, the generative approach avoids the blocking artifacts typical of traditional codecs at low bitrates, providing a more visually pleasing result.

Qualitative comparison of reconstructed frames. The VCM approach (right) maintains clearer details and avoids the blocking artifacts seen in traditional compression (left) at low bitrates.

Conclusion

The VCM paradigm demonstrates that sparse motion patterns can serve as a universal language for both machine and human vision. While currently optimized for surveillance-style scenes with static backgrounds, future work will focus on generalizing the architecture to dynamic environments and more complex scene representations.

Summary updated

Abstract

In this paper, we study a new problem arising from the emerging MPEG standardization effort Video Coding for Machine (VCM) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , which aims to bridge the gap between visual feature compression and classical video coding. VCM is committed to address the requirement of compact signal representation for both machine and human vision in a more or less scalable way. To this end, we make endeavors in leveraging the strength of predictive and generative models to support advanced compression techniques for both machine and human vision tasks simultaneously, in which visual features serve as a bridge to connect signal-level and task-level compact representations in a scalable manner. Specifically, we employ a conditional deep generation network to reconstruct video frames with the guidance of learned motion pattern. By learning to extract sparse motion pattern via a predictive model, the network elegantly leverages the feature representation to generate the appearance of to-be-coded frames via a generative model, relying on the appearance of the coded key frames. Meanwhile, the sparse motion pattern is compact and highly effective for high-level vision tasks, e.g. action recognition. Experimental results demonstrate that our method yields much better reconstruction quality compared with the traditional video codecs (0.0063 gain in SSIM), as well as state-of-the-art action recognition performance over highly compressed videos (9.4% gain in recognition accuracy), which showcases a promising paradigm of coding signal for both human and machine vision.
BibTeX
@inproceedings{xia2020emerging,
  author = {Sifeng Xia and Kunchangtai Liang and Wenhan Yang and Ling-Yu Duan and Jiaying Liu},
  title = {{An Emerging Coding Paradigm Vcm: A Scalable Coding Approach Beyond Feature And Signal}},
  booktitle = {ICME},
  year = {2020},
  url = {https://openreview.net/forum?id=5Yyjv-SkdE},
}
Figure