When modeling high-dimensional data, machine learning practitioners frequently encounter a fundamental bottleneck: the curse of dimensionality applied to covariance matrices. A full covariance matrix for a $d$-dimensional feature space requires tracking $O(d^2)$ parameters and incurs an $O(d^3)$ computational cost for operations like inversion or determinant calculation. To circumvent this, a ubiquitous simplification is made across diverse domains: the diagonal covariance assumption. By setting all off-diagonal elements to zero, features are treated as statistically independent.
While this assumption makes algorithms computationally tractable, it introduces a severe structural blind spot. It forces models to view the world through an axis-aligned lens, completely blind to the intricate, directional correlations that characterize real-world data.
The Computational Allure of Feature Independence
The primary driver behind the diagonal covariance assumption is scalability. In online learning frameworks where models must update parameters on the fly, tracking full feature interactions is often prohibitively expensive.
For example, in Confidence-Weighted (CW) learning for natural language processing, Koby Crammer et al. in Multi-Class Confidence Weighted Algorithms. and Avihai Mejer et al. in Confidence in Structured-Prediction Using Confidence-Weighted Models restrict weight uncertainty models to diagonal covariance matrices. This restriction maintains linear efficiency in high-dimensional spaces but explicitly ignores the rich interactions between different input dimensions that could otherwise inform model updates. Similarly, in Bayesian online multi-label classification, Xinhua Zhang et al. utilize diagonal covariance structures in Bayesian Online Learning for Multi-label and Multi-variate Performance Measures to preserve linear computational efficiency, acknowledging that this posterior independence limits the representation of complex data structures.
This trade-off extends to decentralized and private learning regimes where communication bandwidth is the primary constraint. In decentralized brain imaging classification, Noah Lewis et al. employ diagonal covariance matrices in Decentralized Distribution-sampled Classification Models with Application to Brain Imaging to manage the quadratic communication costs associated with high-dimensional feature sets, exchanging network bandwidth for a slight drop in accuracy compared to centralized approaches. In the context of differential privacy, Boel Nelson demonstrates in PLAN: Variance-Aware Private Mean Estimation that estimating variance strictly along individual coordinates simplifies noise allocation but risks suboptimality when the data's principal components do not align neatly with standard coordinate axes.
The Cost of Axis-Aligned Uncertainty
The consequences of ignoring off-diagonal correlations manifest clearly in probabilistic representation learning. When models represent entities, sentences, or images as probability distributions rather than static points, the geometric shape of that distribution matters. A diagonal covariance matrix restricts the distribution's uncertainty to be axis-aligned—it can only expand or contract parallel to the coordinate axes.
Consider sentence and word embeddings. Kevin Gimpel et al. note in Learning Probabilistic Sentence Representations from Paraphrases that restricting probabilistic sentence embeddings to diagonal covariance matrices assumes all dimensions are independent, reducing computational costs but failing to capture the complex interdependencies inherent in linguistic data. A similar constraint is observed in Probabilistic FastText for Multi-Sense Word Embeddings, where isotropic or spherical covariance matrices limit the model's ability to capture directional semantic features.
In information retrieval and multimodal matching, this independence assumption can degrade performance. Hamed Zamani et al. point out in Multivariate Representation Learning for Information Retrieval that probabilistic dense retrieval models often assume a diagonal covariance matrix to remain efficient, which prevents them from capturing complex inter-dimensional correlations. For image-text matching, the authors of Improved Probabilistic Image-Text Representations explain that while diagonal covariance simplifies distance metrics, moving toward full covariance matrices would allow for a much richer representation of ambiguity and noise, albeit at a significant memory overhead.
In signal processing, such as voice conversion, D. Malah et al. demonstrate in Voice Conversion Using GMM with Enhanced Global Variance that assuming statistical independence between spectral frequency bands over-smooths the speech signal, resulting in a muffled sound.
The Theoretical Disconnect
The diagonal covariance assumption is not just an engineering shortcut; it is deeply embedded in machine learning theory. To prove that an algorithm works or to derive exact error bounds, theorists frequently assume that feature vectors have an identity or spherical covariance matrix.
For instance, in high-dimensional analyses of Lasso-based classification, Malik Tiomoko et al. rely on identity covariance matrices in Deciphering Lasso-based Classification Through a Large Dimensional Analysis of the Iterative Soft-Thresholding Algorithm to ensure mathematical tractability via Random Matrix Theory. Similarly, Eyar Azar et al. utilize spherical covariance assumptions in Semi-Supervised Sparse Gaussian Classification: Provable Benefits of Unlabeled Data to prove the benefits of unlabeled data in sparse classification.
However, as Subha Maity et al. argue in How does overparametrization affect performance on minority groups?, these isotropic simplifications do not reflect the highly correlated, non-linear structures found in real-world datasets, potentially limiting the generalizability of theoretical findings. When analyzing ridge regression with correlated samples, Alexander Atanasov highlights in Risk and cross validation in ridge regression with correlated samples that current high-dimensional models often assume feature-wise and sample-wise correlations are independent, necessitating new mathematical frameworks like free probability theory to relax these constraints.
Bridging the Gap: Low-Rank and Structured Alternatives
Recognizing the limitations of the strict diagonal constraint, recent research focuses on finding middle-ground representations that capture correlations without incurring full quadratic costs.
One prominent strategy is the use of mixture models or factor analyzers. Yunsheng Li et al. address the limitations of standard pooling layers—which utilize diagonal covariance structures—by embedding a Mixture of Factor Analyzers into deep networks in Deep Scene Image Classification with the MFAFVNet, allowing the model to capture high-dimensional visual correlations in scenes. For clustering and parameter recovery, Bingni Guo et al. explore how high-order tensor decompositions can recover diagonal Gaussian Mixture Models in Diagonal Gaussian Mixture Models and Higher Order Tensor Decompositions, though they note that extending these methods to non-diagonal structures remains a critical frontier.
Other approaches modify the underlying architecture to handle structured dependencies. In time-series forecasting, Dongbin Kim et al. point out in Local Geometry Attention for Time Series Forecasting under Realistic Corruptions that approximating a metric tensor as a diagonal matrix fails to capture cross-feature correlations under noise, motivating the need for computationally tractable matrix approximations. In deep self-distillation networks, Etai Littwin et al. analyze the implicit bias of architectures under the assumption of simultaneously diagonalizable covariance matrices in How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks, highlighting the ongoing effort to align theoretical abstractions with natural data distributions.
Key Foundational Work
Constructing informative priors using transfer learning (2006) — contributed 1 concept elements to this direction.
Batch-RLVQ (2002) — contributed 1 concept elements to this direction.