What is the gradient of a scalar function of a symmetric matrix ?

Shriram Srinivasan, Nishant PandaarXiv 2019 · arXiv

Introduction

Matrix calculus is a foundational tool across machine learning, control theory, and statistics. A recurring challenge for researchers is calculating the gradient of a scalar function when the input is a symmetric matrix, such as a covariance matrix. Historically, two conflicting formulas have emerged:

  1. The Standard Mathematical Approach: Based on the Fréchet derivative, this approach yields $G_{sym} = (G + G^T)/2$.
  2. The "Symmetric Gradient" Approach: Popularized in applied fields and references like The Matrix Cookbook, this yields $G_s = G + G^T - G \circ I$.

These formulas are inconsistent, leading to significant confusion and incorrect optimization routines. This paper rigorously proves that the "symmetric gradient" formula is mathematically incorrect due to a flawed mapping between vector spaces.

Methodology: Mapping Between Vector Spaces

The authors frame the problem within finite-dimensional inner-product spaces, using Fréchet derivatives and the Riesz Representation Theorem to define the gradient. To identify the source of the discrepancy, they analyze the mappings between the space of symmetric matrices ($\mathbb{S}^{n \times n}$) and reduced-dimension Euclidean spaces ($\mathbb{R}^m$, where $m = n(n+1)/2$).

Commutative diagram illustrating the mappings between the space of symmetric matrices and reduced-dimension Euclidean spaces.

The analysis utilizes operators such as vec (vectorization), vech (half-vectorization), and the duplication matrix $D$, which maps unique elements back to the full vector space. By tracking these transformations, the authors pinpoint exactly where historical derivations went wrong.

Key Insights: The Root of the Error

The incorrect formula ($G_s = G + G^T - G \circ I$) stems from a fundamental misunderstanding of how to "lift" a gradient from a lower-dimensional space of unique elements back to the matrix space.

Historically, researchers computed the gradient in the reduced space $\mathbb{R}^{n(n+1)/2}$ and naively reshaped it. This process ignores the underlying geometry of the space. To correctly preserve the Frobenius inner product during this transformation, one must include a metric adjustment matrix, $(D^T D)^{-1}$. Without this adjustment, the resulting "gradient" does not satisfy the fundamental properties required by the Riesz Representation Theorem.

Results and Reconciliation

The paper provides a rigorous proof that the only correct symmetric gradient is the symmetric part of the standard gradient:

$$G_{sym} = \text{sym}(G) = \frac{G + G^T}{2}$$

The authors also revisit and debunk classic examples, such as Geering’s 1976 derivation, showing how they violate the chain rule. Crucially, they demonstrate that if the correct metric transformation $(D^T D)^{-1}$ is applied to the reduced-space gradient, it simplifies perfectly to $(G + G^T)/2$. Thus, both conceptual approaches—when mathematically sound—converge on the same simple result.

Conclusion

This work clarifies a long-standing ambiguity in matrix calculus. The primary takeaway is that special, complex formulas are unnecessary for symmetric matrices; the standard projection onto the symmetric subspace is sufficient. While this simplifies theoretical work, practitioners using optimization software must still be careful when mapping 1D arrays back to symmetric matrices. This rigorous projection approach can be extended to other linear manifolds, such as skew-symmetric or triangular matrices, though it does not apply to non-linear manifolds like orthogonal matrices.

Summary updated

Abstract

Perusal of research articles that deal with the topic of matrix calculus reveal two different approaches to calculation of the gradient of a real-valued function of a symmetric matrix leading to two different results. In the mechanics and physics communities, the gradient is calculated using the definition of a \frechet derivative, irrespective of whether the argument is symmetric or not. However, members of the statistics, economics, and electrical engineering communities use another notion of the gradient that explicitly takes into account the symmetry of the matrix, and this "symmetric gradient" $G_s$ is reported to be related to the gradient $G$ computed from the \frechet derivative with respect to a general matrix as $G_s = G + G^T - G \circ I$, where $\circ$ denotes the elementwise Hadamard product of the two matrices. We demonstrate that this relation is incorrect, and reconcile both these viewpoints by proving that $G_s = \mathrm{sym}(G)$.
BibTeX
@misc{srinivasan2019what,
  author = {Shriram Srinivasan and Nishant Panda},
  title = {{What is the gradient of a scalar function of a symmetric matrix ?}},
  year = {2019},
  eprint = {1911.06491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.06491},
}
Figure