Introduction
Matrix calculus is a foundational tool across machine learning, control theory, and statistics. A recurring challenge for researchers is calculating the gradient of a scalar function when the input is a symmetric matrix, such as a covariance matrix. Historically, two conflicting formulas have emerged:
- The Standard Mathematical Approach: Based on the Fréchet derivative, this approach yields $G_{sym} = (G + G^T)/2$.
- The "Symmetric Gradient" Approach: Popularized in applied fields and references like The Matrix Cookbook, this yields $G_s = G + G^T - G \circ I$.
These formulas are inconsistent, leading to significant confusion and incorrect optimization routines. This paper rigorously proves that the "symmetric gradient" formula is mathematically incorrect due to a flawed mapping between vector spaces.
Methodology: Mapping Between Vector Spaces
The authors frame the problem within finite-dimensional inner-product spaces, using Fréchet derivatives and the Riesz Representation Theorem to define the gradient. To identify the source of the discrepancy, they analyze the mappings between the space of symmetric matrices ($\mathbb{S}^{n \times n}$) and reduced-dimension Euclidean spaces ($\mathbb{R}^m$, where $m = n(n+1)/2$).
The analysis utilizes operators such as vec (vectorization), vech (half-vectorization), and the duplication matrix $D$, which maps unique elements back to the full vector space. By tracking these transformations, the authors pinpoint exactly where historical derivations went wrong.
Key Insights: The Root of the Error
The incorrect formula ($G_s = G + G^T - G \circ I$) stems from a fundamental misunderstanding of how to "lift" a gradient from a lower-dimensional space of unique elements back to the matrix space.
Historically, researchers computed the gradient in the reduced space $\mathbb{R}^{n(n+1)/2}$ and naively reshaped it. This process ignores the underlying geometry of the space. To correctly preserve the Frobenius inner product during this transformation, one must include a metric adjustment matrix, $(D^T D)^{-1}$. Without this adjustment, the resulting "gradient" does not satisfy the fundamental properties required by the Riesz Representation Theorem.
Results and Reconciliation
The paper provides a rigorous proof that the only correct symmetric gradient is the symmetric part of the standard gradient:
$$G_{sym} = \text{sym}(G) = \frac{G + G^T}{2}$$
The authors also revisit and debunk classic examples, such as Geering’s 1976 derivation, showing how they violate the chain rule. Crucially, they demonstrate that if the correct metric transformation $(D^T D)^{-1}$ is applied to the reduced-space gradient, it simplifies perfectly to $(G + G^T)/2$. Thus, both conceptual approaches—when mathematically sound—converge on the same simple result.
Conclusion
This work clarifies a long-standing ambiguity in matrix calculus. The primary takeaway is that special, complex formulas are unnecessary for symmetric matrices; the standard projection onto the symmetric subspace is sufficient. While this simplifies theoretical work, practitioners using optimization software must still be careful when mapping 1D arrays back to symmetric matrices. This rigorous projection approach can be extended to other linear manifolds, such as skew-symmetric or triangular matrices, though it does not apply to non-linear manifolds like orthogonal matrices.
Summary updated