Abstract
Video Coding for Machine (VCM) is an emerging paradigm that bridges the gap between human-centric video compression and AI-centric feature extraction. By using sparse motion patterns as a "bridge," this paper proposes a scalable framework that achieves superior action recognition at 1/3 the bitrate of HEVC while providing higher-quality video reconstruction for human viewers.
Introduction
Traditional video codecs, such as H.264 (AVC) and H.265 (HEVC), are primarily optimized for human perception, focusing on minimizing signal-level distortion. However, in modern applications like smart cities and big data analytics, the majority of video content is consumed by machines for tasks such as action recognition and event detection.
Current compression strategies face a significant trade-off: human-centric compression often introduces artifacts that hinder AI performance, while machine-centric feature extraction typically discards the signal data necessary for human verification. This paper introduces a scalable VCM approach that satisfies both requirements by layering machine-vision features with signal-reconstruction data.
Proposed Methodology
The proposed framework integrates feature and video compression through two core components: the Sparse Point Prediction Network (SPPN) and the Motion Guided Generation Network (MGGN).
Sparse Point Prediction Network (SPPN)
Instead of transmitting dense pixel data, the encoder identifies a "sparse motion pattern." Guided by human skeleton data, the SPPN extracts key points representing object movement. Each point is represented by its coordinates and a covariance matrix, resulting in an extremely compact representation (6 float numbers per point).
Motion Guided Generation Network (MGGN)
At the decoder, the MGGN—a conditional deep generative model—uses the reconstructed keyframes and sparse motion patterns to synthesize intermediate frames. It estimates motion flow and deforms keyframe features to match the target pose. This generative approach differs from traditional predictive coding by "hallucinating" realistic pixels based on learned patterns.
Experimental Evaluation
The framework was evaluated using the PKU-MMD dataset, which provides 3D human action data. The model was trained on 3,317 clips and tested on 227 clips, using ground-truth skeleton joints to supervise the SPPN.
Machine Vision Performance
The VCM approach significantly outperformed HEVC in action recognition tasks. Specifically, the proposed method achieved 74.6% accuracy at 5.2 Kbps, whereas HEVC achieved only 65.2% accuracy at a much higher bitrate of 16.2 Kbps. This represents a 9.4% accuracy gain at approximately one-third the bitrate.
Human Vision Performance
For video reconstruction, the VCM method achieved an SSIM of 0.9071 at ~33 Kbps, surpassing HEVC’s 0.9008. Qualitatively, the generative approach avoids the blocking artifacts typical of traditional codecs at low bitrates, providing a more visually pleasing result.
Conclusion
The VCM paradigm demonstrates that sparse motion patterns can serve as a universal language for both machine and human vision. While currently optimized for surveillance-style scenes with static backgrounds, future work will focus on generalizing the architecture to dynamic environments and more complex scene representations.
Summary updated