Associations among Image Assessments as Cost Functions in Linear Decomposition: MSE, SSIM, and Correlation Coefficient

Jianji Wang, Nanning Zheng, Badong Chen, José C. PríncipearXiv 2017 · arXiv

Abstract

In image processing and computer vision, the choice of a cost function significantly influences the outcome of image reconstruction and decomposition tasks. While Mean Squared Error (MSE) is the standard metric for optimization, the Structural Similarity Index (SSIM) and Pearson’s Correlation Coefficient (PCC) are often preferred for their alignment with human visual perception. This paper provides a rigorous mathematical analysis of these three metrics within the framework of linear decomposition. The authors demonstrate that MSE, SSIM, and PCC are mathematically equivalent regarding the selection of basis vectors. Furthermore, they establish that the coefficients derived from these different metrics are related by a simple constant—the correlation coefficient itself—revealing a deep structural link between pixel-wise error and structural similarity.

Introduction

Measuring the fidelity of a reconstructed image is a fundamental challenge in image processing. Traditionally, Mean Squared Error (MSE) has been the dominant metric due to its mathematical simplicity and ease of optimization. However, MSE often fails to capture structural distortions that are obvious to the human eye. To address this, metrics like the Structural Similarity Index (SSIM) were developed to account for luminance, contrast, and structural information. Pearson’s Correlation Coefficient (PCC) serves as another statistical measure of similarity, focusing on the linear relationship between pixel sets.

Despite their conceptual differences, these metrics are frequently used interchangeably as cost functions in optimization problems, such as sparse coding or dictionary learning. This paper investigates whether switching from an MSE-based cost function to an SSIM-based one fundamentally alters the features (basis vectors) selected to represent an image.

Theoretical Framework

The study focuses on linear decomposition, where a target image vector $y$ is approximated by a linear combination of basis vectors $x_i$ from a dictionary, plus an offset $o$:

$$x = s_1x_1 + s_2x_2 + \dots + s_nx_n + o\mathbf{1}$$

Here, $x$ represents the approximation, $s_i$ are the coefficients (or weights), and $o$ is the DC component (offset). The authors analyze three distinct optimization schemes:

  1. MSE Minimization: Minimizing the squared Euclidean distance between $y$ and $x$.
  2. PCC Maximization: Maximizing the absolute value of the correlation coefficient $|r_{xy}|$.
  3. SSIM Maximization: Maximizing the structural similarity index $|SSIM(x, y)|$.

By deriving the partial derivatives for the coefficients $s_i$ and the offset $o$ for each metric, the authors compare the conditions required for optimality.

The Four Associations

The core contribution of the paper is the formalization of four mathematical associations that bridge these metrics:

  1. Equivalence of Optimization Goals: The authors prove that minimizing MSE, maximizing $|r_{xy}|$, and maximizing $|SSIM|$ are all mathematically equivalent to maximizing the absolute value of the correlation coefficient within their respective linear schemes.
  2. Identity of Basis Selection: In a greedy selection process (such as Orthogonal Matching Pursuit), all three metrics will select the exact same basis vectors from a dictionary to represent a target image. This implies that the "features" identified by the algorithm are independent of the choice between MSE and SSIM.
  3. Metric Convergence: The paper demonstrates that the act of minimizing MSE is functionally equivalent to maximizing $|SSIM|$ and $|r_{xy}|$ in the context of linear approximation.
  4. The Coefficient Ratio: While the selected bases are the same, the weights ($s_i$) assigned to them differ. The authors found a constant ratio between the MSE-based coefficients and the SSIM-based coefficients: $$\frac{s_i^{MSE}}{s_i^{SSIM}} = r_{xy}$$

Key Insights: Why SSIM Outperforms MSE

A significant finding of this work explains why SSIM-optimized reconstructions often appear superior to those optimized for MSE. In an MSE-based reconstruction, the variance of the approximation is limited to the covariance between the target and the reconstruction. This often results in a "muted" or blurred appearance because the variance of the output is lower than the variance of the source.

In contrast, SSIM-based optimization forces the variance of the approximation to match the variance of the target image exactly ($\sigma_x^2 = \sigma_y^2$). By preserving the original signal's variance, SSIM maintains the contrast and sharpness that MSE-based methods tend to suppress.

Limitations and Scope

This research is a purely mathematical proof and does not rely on empirical testing against specific datasets (e.g., ImageNet), as the derivations hold for any vector-based linear decomposition. However, several constraints apply:

  • Non-Zero Variance: The proofs assume that image blocks have non-zero variance. In perfectly flat regions (constant pixel values), the denominators in SSIM and PCC formulas become zero, causing the associations to break down.
  • Linearity: The findings are strictly limited to linear transformations. It remains to be seen if these associations persist in non-linear models, such as deep neural networks or non-linear manifold learning.

Conclusion

This paper provides a theoretical unification of MSE, SSIM, and PCC in the context of linear image decomposition. By proving that these metrics lead to the selection of identical basis vectors and establishing a clear mathematical relationship between their optimal coefficients, the authors offer a new perspective on image quality assessment. These insights not only clarify the relationship between pixel-wise error and structural similarity but also suggest potential applications in optimizing data storage and sparse representation algorithms.

Summary updated

Abstract

The traditional methods of image assessment, such as mean squared error (MSE), signal-to-noise ratio (SNR), and Peak signal-to-noise ratio (PSNR), are all based on the absolute error of images. Pearson's inner-product correlation coefficient (PCC) is also usually used to measure the similarity between images. Structural similarity (SSIM) index is another important measurement which has been shown to be more effective in the human vision system (HVS). Although there are many essential differences among these image assessments, some important associations among them as cost functions in linear decomposition are discussed in this paper. Firstly, the selected bases from a basis set for a target vector are the same in the linear decomposition schemes with different cost functions MSE, SSIM, and PCC. Moreover, for a target vector, the ratio of the corresponding affine parameters in the MSE-based linear decomposition scheme and the SSIM-based scheme is a constant, which is just the value of PCC between the target vector and its estimated vector.
BibTeX
@misc{wang2017associations,
  author = {Jianji Wang and Nanning Zheng and Badong Chen and José C. Príncipe},
  title = {{Associations among Image Assessments as Cost Functions in Linear Decomposition: MSE, SSIM, and Correlation Coefficient}},
  year = {2017},
  eprint = {1708.01541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.01541},
}
Figure