Perceptual Video Coding for Machines via Satisfied Machine Ratio Modeling

Qi Zhang, Shanshe Wang, Xinfeng Zhang, Chuanmin Jia, Zhao Wang, Siwei Ma, Wen Gao

TL;DR

This paper introduces Satisfied Machine Ratio (SMR), a novel statistical metric designed to evaluate the perceptual quality of compressed images and videos specifically for computer vision models rather than human eyes. By modeling the diverse ways different AI models tolerate compression artifacts, the authors develop an SMR prediction framework that achieves up to 50% bit-rate savings while maintaining robust performance across unseen machines, datasets, and video codecs.

What Problem Does This Paper Address?

Video Coding for Machines (VCM) is an emerging field dedicated to compressing visual data for AI consumption (like object detectors and image classifiers) rather than human viewing. However, existing VCM methods suffer from a major flaw: they usually optimize for just one or two specific models.

In the real world, "machine diversity" is a massive hurdle. Different neural network architectures (e.g., CNNs vs. Transformers) perceive compression distortions differently. As the authors demonstrate, optimizing a video encoder to satisfy one specific model can actively degrade the performance of another.

Fig. 1. The diversity scores of pairs of machines measured by the average Hamming distance of machine perception consistency sequences.

Furthermore, traditional compression metrics optimize for the Human Vision System (HVS), ignoring the unique "perceptual" characteristics of machines. This paper bridges the gap by shifting VCM from model-specific optimization to a generalized, population-based approach.

Methods

The authors propose Satisfied Machine Ratio (SMR), inspired by the human Satisfied User Ratio (SUR) metric. SMR calculates the proportion of models in a diverse "machine library" that remain "satisfied" with an image's quality at a given compression level. "Satisfaction" means the model's output on the compressed image matches its output on the uncompressed original (e.g., maintaining top-K classification accuracy or object detection mAP).

To make this practical without running dozens of models during encoding, they design a deep learning-based SMR Prediction Model:

  • Siamese Feature Extraction: A shared EfficientNet-B4 backbone extracts high-level semantic features from both the original and compressed images.
  • MLP Regressor: The concatenated features are passed to a Multi-Layer Perceptron to predict the continuous SMR value.
  • Auxiliary Task: To boost accuracy, they introduce a pairwise training task that forces the network to predict the difference in SMR between two differently compressed versions of the same image.

Datasets

To train and validate their models, the authors created a massive, first-of-its-kind SMR dataset based on MS COCO 2017, compressed across 36 different Quantization Parameters (QPs) using HEVC:

  • Image Classification Library: Features 58 models (v1) and 72 models (v2), generating over 22.8 million images and 137 million SMR labels.
  • Object Detection Library: Features 98 distinct models, generating 4.5 million full images and over 456 million SMR labels.

To test generalizability, they evaluated their framework on unseen datasets, including PASCAL VOC test2007, the TVD dataset, and the SFU-HW dataset for video inter-frame testing.

Key Insights and Ideas

  • Machines Don't Have a Monotonic JND: Humans have a strict "Just Noticeable Difference" (JND) threshold below which distortion is invisible. Machines do not. A higher compression rate (higher QP) can sometimes paradoxically increase a machine's prediction consistency, leading to non-monotonic SMR curves.
  • SMR vs. SUR: Human satisfaction (SUR) drops off precipitously at mid-range compression levels. In contrast, machine satisfaction (SMR) degrades much more gradually, proving that machines are far more resilient to heavy distortion but highly sensitive to subtle, low-level pixel changes.
  • Generalized Perceptual Features: Deep feature distances (like cosine distance) correlate strongly but non-linearly with machine satisfaction, making deep neural networks excellent proxies for predicting SMR.
Fig. 3. QP-SMR curves of several randomly selected images. The corresponding image (or object image) and its compressed versions are displayed under the curve (better viewed by zooming in), where the leftmost one is the original object image, and the rest are compressed variants with QP = 32, 40, 45, 48, and 51, respectively. The top 4 sub-figures are for image classification, and the bottom 4 are for object detection.

Key Results

Using SMR to guide compression parameters yields massive efficiency gains (measured via BD-rate savings, where lower is better):

  • Image Classification: Ground-truth SMR optimization achieved 39.2% to 50.6% BD-rate savings over standard HEVC. The SMR prediction model retained most of this benefit, delivering 30.4% to 44.6% savings.
  • Object Detection: Achieved ~23% BD-rate savings using ground-truth SMR and ~10-12% savings using the predicted SMR.
  • Cross-Codec Generalization: Even though the SMR predictor was trained purely on HEVC data, it successfully generalized to state-of-the-art traditional codecs (VVC and AVS3), neural compression methods, and specialized MVS-oriented codecs.

Limitations and Future Work

The primary limitation acknowledged by the authors is that their current SMR model is full-reference, meaning it requires the original, uncompressed image to predict the feature differences. In many real-world streaming or storage pipelines, the uncompressed original is unavailable. The authors note that developing a no-reference SMR model is a crucial next step. Additionally, while they tested their model on video inter-frames (P-frames and B-frames), directly integrating SMR into the frame-by-frame rate-distortion optimization loop of video encoders remains an open challenge for future research.

Summary updated

Abstract

Video Coding for Machines (VCM) aims to compress visual signals for machine analysis. However, existing methods only consider a few machines, neglecting the majority. Moreover, the machine's perceptual characteristics are not leveraged effectively, resulting in suboptimal compression efficiency. To overcome these limitations, this paper introduces Satisfied Machine Ratio (SMR), a metric that statistically evaluates the perceptual quality of compressed images and videos for machines by aggregating satisfaction scores from them. Each score is derived from machine perceptual differences between original and compressed images. Targeting image classification and object detection tasks, we build two representative machine libraries for SMR annotation and create a large-scale SMR dataset to facilitate SMR studies. We then propose an SMR prediction model based on the correlation between deep feature differences and SMR. Furthermore, we introduce an auxiliary task to increase the prediction accuracy by predicting the SMR difference between two images in different quality. Extensive experiments demonstrate that SMR models significantly improve compression performance for machines and exhibit robust generalizability on unseen machines, codecs, datasets, and frame types. SMR enables perceptual coding for machines and propels VCM from specificity to generality. Code is available at https://github.com/ywwynm/SMR.
BibTeX
@article{zhang2024perceptual,
  author = {Qi Zhang and Shanshe Wang and Xinfeng Zhang and Chuanmin Jia and Zhao Wang and Siwei Ma and Wen Gao},
  title = {{Perceptual Video Coding for Machines via Satisfied Machine Ratio Modeling}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3393633},
}
Figure