A Hitchhiker's Guide to Structural Similarity

Abhinau K. Venkataramanan, Chengyang Wu, Alan C. Bovik, Ioannis Katsavounidis, Zafar Shahid

Abstract

The Structural Similarity (SSIM) Index is a very widely used image/video quality model that continues to play an important role in the perceptual evaluation of compression algorithms, encoding recipes and numerous other image/video processing algorithms. Several public implementations of the SSIM and Multiscale-SSIM (MS-SSIM) algorithms have been developed, which differ in efficiency and performance. This “bendable ruler” makes the process of quality assessment of encoding algorithms unreliable. To address this situation, we studied and compared the functions and performances of popular and widely used implementations of SSIM, and we also considered a variety of design choices. Based on our studies and experiments, we have arrived at a collection of recommendations on how to use SSIM most effectively, including ways to reduce its computational burden.

Introduction: The "Bendable Ruler" Problem

In the competitive landscape of video streaming—led by giants like Netflix, YouTube, and Facebook—encoding algorithms must strike a delicate balance between file size and visual fidelity. The Structural Similarity (SSIM) Index has long served as the industry's primary metric for measuring perceptual image quality. However, a significant issue persists: SSIM is not a single, monolithic algorithm.

Different software libraries, such as FFmpeg, MATLAB, and OpenCV, employ varying default window functions, scaling factors, and parameters. This inconsistency effectively turns SSIM into a "bendable ruler," making it nearly impossible for engineers to reliably compare results across different platforms. Furthermore, as video resolutions climb to 4K and beyond, the computational overhead of calculating SSIM becomes a major bottleneck in Rate-Distortion Optimization (RDO) pipelines. This research seeks to standardize SSIM usage and provide high-performance alternatives for modern hardware.

Methodology and Algorithmic Optimization

The researchers conducted a comprehensive investigation into the efficiency and accuracy of existing SSIM implementations through several key approaches:

  1. Benchmarking: The study compared 12 SSIM and 8 Multi-Scale SSIM (MS-SSIM) implementations to identify "Pareto-optimal" versions that best balance accuracy (correlation with human scores) and execution time.
  2. Computational Efficiency: The authors demonstrated that using integral images with rectangular windows reduces computational complexity from $O(MNk^2)$ to $O(MN)$. This shift makes the processing time independent of the window size, a vital optimization for high-resolution content.
  3. Scaled SSIM: To alleviate the burden on encoding pipelines, the study proposed models using Histogram Matching and Neural Networks to predict high-resolution SSIM scores using only low-resolution data proxies.
  4. Enhanced SSIM: A new "best-of-breed" implementation was proposed, utilizing a stride of 5 (for a 25x speedup), rectangular windows, and spatial pooling via the Coefficient of Variation (CoV).

Datasets and Evaluation

To ensure the findings were grounded in human perception, the study utilized four major public databases:

  • LIVE IQA & TID2013: Standard benchmarks for static image quality.
  • LIVE VQA: A video quality database featuring various distortion types.
  • Netflix Public Database: Real-world streaming data used for the VMAF project.
  • In-house Corpus: 60 pristine videos compressed at various scales and bitrates, specifically used to train the "Scaled SSIM" models.

Key Insights and Recommendations

  • Rectangular vs. Gaussian Windows: Surprisingly, the study found that simple rectangular windows are objectively superior to the traditionally recommended Gaussian windows. They offer slightly higher peak performance and allow for massive speedups via integral images.
  • The Power of Striding: Computing SSIM on every 5th pixel (a stride of 5) results in a 25x speedup with almost zero impact on accuracy, providing a significant win for real-time processing.
  • Device-Dependent Metrics: The paper emphasizes that SSIM should be "device-dependent." By down-sampling images based on the intended viewing distance (e.g., TV vs. Mobile), the metric better reflects actual human perception.
  • Color Awareness: While most SSIM use is luminance-only, the authors show that Quaternion SSIM (treating RGB as a vector) and YUV-weighted models provide a noticeable boost in predicting human quality scores.

Results and Performance

The benchmarking identified LIBVMAF, Daala, and FastQA as the most efficient and accurate off-the-shelf implementations. The proposed "Enhanced SSIM" matched or exceeded the accuracy of LIBVMAF while being significantly faster due to striding and integral image optimizations. Furthermore, the Histogram Matching approach for Scaled SSIM achieved near-perfect correlation (0.99+) with "true" SSIM scores, proving that 4K quality can be accurately estimated from much smaller, faster-to-process proxies.

Conclusion and Future Directions

While the study provides a robust framework for standardized SSIM, limitations remain. The 5-parameter logistic (5PL) mapping used to improve correlation with human Mean Opinion Scores (MOS) may not generalize perfectly across all content types, such as social media user-generated content.

Future work will focus on temporal pooling to better capture "visual fatigue" and the impact of transient glitches in long-form video. Additionally, integrating these optimized SSIM variants directly into hardware encoders is a promising next step for the industry, ensuring that perceptual quality remains at the forefront of video delivery technology.

Summary updated

Abstract

The Structural Similarity (SSIM) Index is a very widely used image/video quality model that continues to play an important role in the perceptual evaluation of compression algorithms, encoding recipes and numerous other image/video processing algorithms. Several public implementations of the SSIM and Multiscale-SSIM (MS-SSIM) algorithms have been developed, which differ in efficiency and performance. This “bendable ruler” makes the process of quality assessment of encoding algorithms unreliable. To address this situation, we studied and compared the functions and performances of popular and widely used implementations of SSIM, and we also considered a variety of design choices. Based on our studies and experiments, we have arrived at a collection of recommendations on how to use SSIM most effectively, including ways to reduce its computational burden.
BibTeX
@inproceedings{venkataramanan2021hitchhiker,
  author = {Abhinau K. Venkataramanan and Chengyang Wu and Alan C. Bovik and Ioannis Katsavounidis and Zafar Shahid},
  title = {{A Hitchhiker's Guide to Structural Similarity}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=IHlkwUT_S99},
}
Figure