A Hitchhiker's Guide to Structural Similarity

Abhinau K. Venkataramanan, Chengyang Wu, Alan C. Bovik, Ioannis Katsavounidis, Zafar Shahid

Abstract

The Structural Similarity (SSIM) Index is a very widely used image/video quality model that continues to play an important role in the perceptual evaluation of compression algorithms, encoding recipes and numerous other image/video processing algorithms. Several public implementations of the SSIM and Multiscale-SSIM (MS-SSIM) algorithms have been developed, which differ in efficiency and performance. This "bendable ruler" makes the process of quality assessment of encoding algorithms unreliable. To address this situation, we studied and compared the functions and performances of popular and widely used implementations of SSIM, and we also considered a variety of design choices. Based on our studies and experiments, we have arrived at a collection of recommendations on how to use SSIM most effectively, including ways to reduce its computational burden.

Introduction: The "Bendable Ruler" Problem

In the field of video compression, the Structural Similarity (SSIM) index is the industry standard for measuring perceptual quality loss. However, a significant challenge persists: the lack of standardization across public implementations. Depending on the software library used—such as FFmpeg, MATLAB, or Scikit-image—the resulting scores for the same video can vary. This "bendable ruler" phenomenon undermines the reliability of quality assessment and makes cross-platform comparisons difficult. Furthermore, as video resolutions climb to 4K and 8K, the computational cost of calculating SSIM becomes a bottleneck in Rate-Distortion Optimization (RDO).

Methodology

The researchers conducted a comprehensive analysis of ten SSIM and eight MS-SSIM (Multi-Scale) implementations. The study focused on identifying optimal design choices to balance accuracy and computational efficiency through several approaches:

  • Ablation Studies: Testing the impact of window functions (Gaussian vs. Rectangular), window sizes, and pixel "stride" (skipping pixels to save time).
  • Computational Optimization: The use of summed-area tables (integral images) was proposed to reduce the complexity of rectangular window calculations from $O(MNk^2)$ to $O(MN)$, making the process independent of window size.
  • Scaled SSIM: Development of models, including Neural Networks and Support Vector Regressors, to predict high-resolution SSIM scores using only low-resolution data.
  • Enhanced SSIM: A new, open-source variant integrating rectangular windows, a stride of 5, and optimized spatial-temporal pooling methods.

To ensure robustness, these methods were validated against four major benchmarks: the LIVE Image Quality Assessment (IQA) Database, the Tampere Image Database 2013 (TID2013), the LIVE Video Quality Assessment (VQA) Database, and the Netflix Public Database.

Key Design Insights and Optimization

Contrary to long-standing advice in image processing to use Gaussian windows to avoid "noise leakage," the study found that rectangular windows actually outperform Gaussian windows for smaller window sizes (linear dimensions of 15–20). This is a significant finding because rectangular windows are much faster to compute using integral images.

Additionally, the research demonstrated that a stride of 5 (calculating every 5th pixel) yields a 25x speedup with almost no change in prediction accuracy. The inclusion of color information also proved vital; incorporating color via Quaternion SSIM in YUV space significantly boosted the correlation with human subjective scores compared to traditional luminance-only (Y) implementations.

Performance Evaluation

The proposed Enhanced SSIM variant was evaluated against benchmarks like the Netflix Public Database, which focuses on spatial scaling and compression. Enhanced SSIM achieved a Spearman Rank Order Correlation Coefficient (SROCC) of 0.8360, significantly outperforming the widely used LIBVMAF implementation, which scored 0.7652.

The study also identified a "Pareto front" of implementations, highlighting Daala’s Fast MS-SSIM and Scikit-Video’s SSIM as offering the best trade-offs between execution time and accuracy. These findings provide a clear roadmap for developers to choose the most efficient implementation for their specific needs.

Conclusion

Standardizing SSIM is crucial for consistent video quality assessment across the industry. By adopting optimized design choices like rectangular windows and strategic striding, practitioners can achieve high-fidelity quality metrics with significantly reduced computational overhead. While cross-database generalization remains a challenge for mapping functions, the move toward an "Enhanced SSIM" standard offers a more reliable and efficient "ruler" for the next generation of video processing algorithms.

Summary updated

Abstract

The Structural Similarity (SSIM) Index is a very widely used image/video quality model that continues to play an important role in the perceptual evaluation of compression algorithms, encoding recipes and numerous other image/video processing algorithms. Several public implementations of the SSIM and Multiscale-SSIM (MS-SSIM) algorithms have been developed, which differ in efficiency and performance. This "bendable ruler" makes the process of quality assessment of encoding algorithms unreliable. To address this situation, we studied and compared the functions and performances of popular and widely used implementations of SSIM, and we also considered a variety of design choices. Based on our studies and experiments, we have arrived at a collection of recommendations on how to use SSIM most effectively, including ways to reduce its computational burden.
BibTeX
@inproceedings{venkataramanan2021hitchhiker,
  author = {Abhinau K. Venkataramanan and Chengyang Wu and Alan C. Bovik and Ioannis Katsavounidis and Zafar Shahid},
  title = {{A Hitchhiker's Guide to Structural Similarity}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=IHlkwUT_S99},
}
Figure