TL;DR
This paper introduces Satisfied Machine Ratio (SMR), a novel statistical metric designed to evaluate the perceptual quality of compressed images and videos specifically for computer vision models rather than human eyes. By modeling the diverse ways different AI models tolerate compression artifacts, the authors develop an SMR prediction framework that achieves up to 50% bit-rate savings while maintaining robust performance across unseen machines, datasets, and video codecs.
What Problem Does This Paper Address?
Video Coding for Machines (VCM) is an emerging field dedicated to compressing visual data for AI consumption (like object detectors and image classifiers) rather than human viewing. However, existing VCM methods suffer from a major flaw: they usually optimize for just one or two specific models.
In the real world, "machine diversity" is a massive hurdle. Different neural network architectures (e.g., CNNs vs. Transformers) perceive compression distortions differently. As the authors demonstrate, optimizing a video encoder to satisfy one specific model can actively degrade the performance of another.
Furthermore, traditional compression metrics optimize for the Human Vision System (HVS), ignoring the unique "perceptual" characteristics of machines. This paper bridges the gap by shifting VCM from model-specific optimization to a generalized, population-based approach.
Methods
The authors propose Satisfied Machine Ratio (SMR), inspired by the human Satisfied User Ratio (SUR) metric. SMR calculates the proportion of models in a diverse "machine library" that remain "satisfied" with an image's quality at a given compression level. "Satisfaction" means the model's output on the compressed image matches its output on the uncompressed original (e.g., maintaining top-K classification accuracy or object detection mAP).
To make this practical without running dozens of models during encoding, they design a deep learning-based SMR Prediction Model:
- Siamese Feature Extraction: A shared EfficientNet-B4 backbone extracts high-level semantic features from both the original and compressed images.
- MLP Regressor: The concatenated features are passed to a Multi-Layer Perceptron to predict the continuous SMR value.
- Auxiliary Task: To boost accuracy, they introduce a pairwise training task that forces the network to predict the difference in SMR between two differently compressed versions of the same image.
Datasets
To train and validate their models, the authors created a massive, first-of-its-kind SMR dataset based on MS COCO 2017, compressed across 36 different Quantization Parameters (QPs) using HEVC:
- Image Classification Library: Features 58 models (v1) and 72 models (v2), generating over 22.8 million images and 137 million SMR labels.
- Object Detection Library: Features 98 distinct models, generating 4.5 million full images and over 456 million SMR labels.
To test generalizability, they evaluated their framework on unseen datasets, including PASCAL VOC test2007, the TVD dataset, and the SFU-HW dataset for video inter-frame testing.
Key Insights and Ideas
- Machines Don't Have a Monotonic JND: Humans have a strict "Just Noticeable Difference" (JND) threshold below which distortion is invisible. Machines do not. A higher compression rate (higher QP) can sometimes paradoxically increase a machine's prediction consistency, leading to non-monotonic SMR curves.
- SMR vs. SUR: Human satisfaction (SUR) drops off precipitously at mid-range compression levels. In contrast, machine satisfaction (SMR) degrades much more gradually, proving that machines are far more resilient to heavy distortion but highly sensitive to subtle, low-level pixel changes.
- Generalized Perceptual Features: Deep feature distances (like cosine distance) correlate strongly but non-linearly with machine satisfaction, making deep neural networks excellent proxies for predicting SMR.
Key Results
Using SMR to guide compression parameters yields massive efficiency gains (measured via BD-rate savings, where lower is better):
- Image Classification: Ground-truth SMR optimization achieved 39.2% to 50.6% BD-rate savings over standard HEVC. The SMR prediction model retained most of this benefit, delivering 30.4% to 44.6% savings.
- Object Detection: Achieved ~23% BD-rate savings using ground-truth SMR and ~10-12% savings using the predicted SMR.
- Cross-Codec Generalization: Even though the SMR predictor was trained purely on HEVC data, it successfully generalized to state-of-the-art traditional codecs (VVC and AVS3), neural compression methods, and specialized MVS-oriented codecs.
Limitations and Future Work
The primary limitation acknowledged by the authors is that their current SMR model is full-reference, meaning it requires the original, uncompressed image to predict the feature differences. In many real-world streaming or storage pipelines, the uncompressed original is unavailable. The authors note that developing a no-reference SMR model is a crucial next step. Additionally, while they tested their model on video inter-frames (P-frames and B-frames), directly integrating SMR into the frame-by-frame rate-distortion optimization loop of video encoders remains an open challenge for future research.
Summary updated