Benchmarking JPEG XL image compression

Thai Duong, Sami Boukortt, Touradj Ebrahimi, Evgenii Kliuchnikov, Jon Sneyers, Evgeniy Upenik, Lode Vandevenne‎, Luca Versari, Jan Wassenberg2020 · DOI

TL;DR

This paper benchmarks the new JPEG XL image compression codec against established standards like JPEG and HEVC. It finds that JPEG XL offers a compelling balance of high visual quality, efficient compression, and fast decoding speeds, making it a strong contender for modern image distribution.

What Problem Does This Paper Address?

The ever-increasing resolution of digital images and the demand for high-quality visual experiences on the web present a significant challenge for image compression. Existing codecs, while effective, often struggle to balance compression efficiency with perceptual visual quality, especially for high dynamic range (HDR) and wide color gamut images. Furthermore, the need for backward compatibility with existing JPEG files and the demand for fast, low-power decoding on various devices add further complexity. This paper aims to provide a comprehensive evaluation of the newly developed JPEG XL codec, comparing its performance against widely used codecs like JPEG and HEVC across subjective and objective quality metrics, as well as encoding and decoding speeds.

Methods

The authors conducted a thorough benchmarking of JPEG XL using a combination of subjective and objective evaluation methods.

  • Subjective Evaluation: They employed the Degradation Category Rating (DCR-HR) method, a standard ITU recommendation, to assess perceptual visual quality. This involved presenting participants with original images alongside compressed versions and asking them to rate the perceived impairment on a five-level scale.
  • Objective Quality Metrics: Two objective metrics were used:
  • Butteraugli: A Google-developed metric that models human visual system properties to assess perceptual quality. It was used to measure both maximum error and a p-norm average.
  • MS-SSIM: A widely used metric that quantifies distortion by considering structural and contrast information in images.
  • Speed Benchmarking: The encoding and decoding speeds of JPEG XL were measured on both x86 and ARM architectures, with varying levels of parallelization. The impact of JPEG XL's "adaptive reconstruction" feature on speed was also investigated.
  • Codec Implementations: The study used specific, well-regarded implementations for each codec: Guetzli and Libjpeg for JPEG, Kakadu for JPEG 2000, and HM and x265 for HEVC.

Datasets

The study utilized a dataset of eight original images, comprising six photographic and two non-photographic images of varying sizes and content. This dataset was used to generate compressed stimuli for both subjective and objective evaluations. The paper also references a larger dataset of 10 images used for objective quality metrics and speed measurements, including a 45.4-megapixel image to test scalability.

Key Insights and Ideas

  • JPEG XL's Dual Value: JPEG XL is designed to provide benefits for both existing JPEG users (through lossless transcoding) and new users seeking advanced compression features.
  • Perceptual Quality Focus: The codec prioritizes perceptual visual quality, aiming for "visually lossless" compression at typical internet bitrates.
  • Speed Advantage: JPEG XL is engineered for efficient parallel processing, leading to faster decoding speeds than even the original JPEG.
  • Lossless Transcoding: A key feature is its ability to reconstruct original JPEGs byte-for-byte, preserving digital heritage without introducing new artifacts.
  • Beyond L2 Norms: The authors advocate for metrics that report maximum error (like Butteraugli's max norm) over L2-norm based metrics (like MS-SSIM), as they provide better confidence that images are not unacceptably degraded anywhere.

Key Results

  • Subjective Quality: At bitrates around 1 bpp, JPEG XL and HEVC are generally indistinguishable from the original. HEVC sometimes produces more visually pleasing images at lower bitrates, but JPEG XL excels at maintaining facial details and texture at higher, good-quality bitrates.
  • Objective Quality (Butteraugli): JPEG XL consistently achieved lower maximum and p-norm Butteraugli distances compared to JPEG, JPEG 2000, and HEVC across all tested bitrates. On average, JPEG XL's Butteraugli distance was 57-69% of other codecs.
  • Objective Quality (MS-SSIM): JPEG XL showed similar performance to HEVC-HM-YUV444, though at slightly higher bitrates. JP2visual often achieved higher MS-SSIM scores.
  • Encoding Speed: JPEG XL's "cheetah" speed setting is significantly faster than HEVC-HM-YUV444 (3 orders of magnitude) and HEVC-x265 (13 times faster) when using 4 threads. The default "squirrel" setting is 3 times faster than HEVC-x265.
  • Decoding Speed: JPEG XL's decoder is 9 times faster than FFmpeg (which is not parallelized) when using 4 threads.
  • Scalability: JPEG XL demonstrates good scalability with increasing thread counts on both large and small images.

Limitations and Future Work

  • Subjective Evaluation Constraints: The laboratory conditions for subjective evaluation (lower screen brightness) might differ from typical home or office usage, potentially affecting how artifacts are perceived.
  • Butteraugli Optimization: The JPEG XL encoder used in the study was not specifically optimized for the Butteraugli metric, suggesting that further improvements might be possible.
  • HEVC Implementation: The HEVC-HM implementation used was noted as not being commonly used in practice due to its low encoding throughput.
  • Future Work: The authors plan to further describe Butteraugli in detail and characterize its performance. They also suggest that future work could explore the impact of adaptive reconstruction at different quality settings.

Summary updated

Abstract

JPEG XL is a practical, royalty-free codec for scalable web distribution and efficient compression of high-quality photographs. It also includes previews, progressiveness, animation, transparency, high dynamic range, wide color gamut, and high bit depth. Users expect faithful reproductions of ever higher-resolution images. Experiments performed during standardization have shown the feasibility of economical storage without perceptible quality loss, lossless transcoding of existing JPEG, and fast software encoders and decoders. We analyse the results of subjective and objective evaluations versus existing codecs including HEVC and JPEG. New image codecs have to co-exist with their previous generations for several years. JPEG XL is unique in providing value for both existing JPEGs as well as new users. It includes coding tools to reduce the transmission and storage costs of JPEG by 22% while allowing byte-for-byte exact reconstruction of the original JPEG. Avoiding transcoding and additional artifacts helps to preserve our digital heritage. Applications require fast and low-power decoding. JPEG XL was designed to benefit from multicore and SIMD, and actually decodes faster than JPEG. We report the resulting speeds on ARM and x86 CPUs. To enable reproduction of these results, we open sourced the JPEG XL software in 2019.
BibTeX
@misc{duong2020benchmarking,
  author = {Thai Duong and Sami Boukortt and Touradj Ebrahimi and Evgenii Kliuchnikov and Jon Sneyers and Evgeniy Upenik and Lode Vandevenne‎ and Luca Versari and Jan Wassenberg},
  title = {{Benchmarking JPEG XL image compression}},
  year = {2020},
  doi = {10.1117/12.2556264},
}
Figure