TL;DR
This paper reveals a significant disconnect between standard image quality metrics (like MS-SSIM) and actual human perception in the context of deep learning-based compression. Through extensive user studies, the authors demonstrate that traditional engineered codecs like BPG often look better to humans than neural models, even when those models achieve much higher metric scores.
What Problem Does This Paper Address?
In the race to develop better image compression, the research community has shifted from hand-engineered algorithms (like JPEG and BPG) toward deep learning models. To evaluate these new models, researchers typically rely on automated metrics: Peak Signal-to-Noise Ratio (PSNR) and Multi-Scale Structural Similarity (MS-SSIM).
Because these metrics are differentiable, many deep learning models are trained to optimize them directly. However, this creates a potential "Goodhart’s Law" scenario: when a measure becomes a target, it ceases to be a good measure. The authors set out to determine if the high MS-SSIM scores touted by neural compression papers actually translate to images that look better to the human eye.
Methods
The authors conducted a large-scale human perceptual study using Amazon Mechanical Turk (MTurk). They compared four compression methods:
- Engineered: JPEG-2000 and BPG.
- Learned: Two prominent deep learning models by Mentzer et al. and Ballé et al.
The evaluation used a 2-Alternative Forced Choice (2AFC) setup. Evaluators were shown the original image alongside two reconstructed versions from different methods and asked to pick the one most similar to the original. To ensure a fair comparison, they developed a custom interface with a synchronized magnifying glass, allowing users to inspect fine details across all three images simultaneously.
The study covered four different compression levels (0.23, 0.37, 0.67, and 1.0 bits-per-pixel) and resulted in over 17,000 individual human judgments.
Datasets
The study utilized four standard benchmarks in the image processing community:
- Kodak: A classic set of 24 high-quality film images.
- Urban100: Images of urban scenes with many repetitive structures.
- Set14 & Set5: Smaller, standard datasets often used in super-resolution and compression tasks.
The deep learning models were trained on the ImageNet dataset.
Key Insights and Ideas
- Metric Blindness: MS-SSIM is surprisingly poor at differentiating between a sharp patch and a locally blurry one. Neural models optimized for MS-SSIM often produce "smeared" or artifact-heavy textures that the metric ignores but humans find distracting.
- The BPG Dominance: Despite having lower MS-SSIM scores than the neural model by Mentzer et al., the engineered BPG codec was overwhelmingly preferred by human evaluators across almost all datasets and bitrates.
- Text Readability: One of the most striking failures of learned models is text reconstruction. As seen in Figure 1, neural models can render text completely unreadable while maintaining high structural similarity scores, whereas traditional methods preserve the legibility of the characters.
Key Results
The quantitative results highlight a massive gap between automated rankings and human preference:
- Kodak Dataset: Mentzer et al. achieved the highest MS-SSIM across all bitrates. However, human studies showed BPG performed significantly better than all other methods.
- Ballé et al. Failure: In the human study, the model by Ballé et al. was never voted the best method for any image, despite often beating BPG on MS-SSIM at higher bitrates.
- PSNR vs. MS-SSIM: When BPG was excluded, JPEG-2000 (an engineered method) was preferred at higher bitrates, even though it had the lowest MS-SSIM. This suggests that for learned approaches, MS-SSIM is a better loss function than PSNR, but it still falls short of human judgment.
Limitations and Future Work
The authors acknowledge that they only evaluated two specific deep learning implementations, largely because many other researchers do not release their code or reconstructed images.
The primary takeaway is a call to action for the field: we need better perceptual metrics. Until a metric exists that truly aligns with human vision, the authors argue that researchers must provide implementations or at least standardized image sets so that qualitative human comparison remains possible. Relying solely on MS-SSIM leads to models that "game the metric" rather than providing better visual quality for end-users.
Summary updated