On estimating covariances between many assets with histories of highly variable length

Robert B. Gramacy, Joo Hee Lee, Ricardo SilvaarXiv 2007 · DOI · arXiv

Abstract

Quantitative portfolio allocation requires the accurate and tractable estimation of covariances between a large number of assets, whose histories can greatly vary in length. Such data are said to follow a monotone missingness pattern, under which the likelihood has a convenient factorization. Upon further assuming that asset returns are multivariate normally distributed, with histories at least as long as the total asset count, maximum likelihood (ML) estimates are easily obtained by performing repeated ordinary least squares (OLS) regressions, one for each asset. Things get more interesting when there are more assets than historical returns. OLS becomes unstable due to rank--deficient design matrices, which is called a "big p small n" problem. We explore remedies that involve making a change of basis, as in principal components or partial least squares regression, or by applying shrinkage methods like ridge regression or the lasso. This enables the estimation of covariances between large sets of assets with histories of essentially arbitrary length, and offers improvements in accuracy and interpretation. We further extend the method by showing how external factors can be incorporated. This allows for the adaptive use of factors without the restrictive assumptions common in factor models. Our methods are demonstrated on randomly generated data, and then benchmarked by the performance of balanced portfolios using real historical financial returns. An accompanying R package called monomvn, containing code implementing the estimators described herein, has been made freely available on CRAN.

Introduction

In quantitative finance, constructing an optimal portfolio necessitates a precise covariance matrix to understand the co-movements of assets. However, real-world financial data is often characterized by varying history lengths—for instance, a mix of established blue-chip stocks with decades of data and recent IPOs with only a few months. Standard statistical approaches typically handle this by either discarding data (the "complete case" approach) or using pairwise estimates that may result in non-positive definite matrices, rendering them useless for optimization.

This paper addresses the "big $p$, small $n$" problem in the context of "monotone missingness," where data follows a staircase-like pattern. While methods exist for this pattern using Ordinary Least Squares (OLS) regression, they fail when the number of assets ($p$) exceeds the number of observations ($n$). The authors propose a robust framework using parsimonious regression techniques to ensure stable and accurate covariance estimation regardless of history length.

Methodology: Recursive Regression and Parsimony

The core of the proposed approach is a recursive regression algorithm that factorizes the likelihood of multivariate normal data. When data exhibits a monotone missingness pattern, the joint distribution can be broken down into a series of conditional distributions. For each asset $j$, a regression is performed using the data available for that asset and all previously processed assets with longer or equal histories.

Illustration of the monotone missingness pattern and the recursive regression step for asset $j=5$. The red box indicates the predictor matrix $\mathbf{Y}_5$ and the green box indicates the response vector $\mathbf{y}_5$.

To overcome the instability of OLS in high-dimensional settings, the authors introduce several parsimonious regression methods:

  • Shrinkage Methods: Ridge regression and the Lasso shrink coefficients toward zero, preventing overfitting. The Lasso also performs automatic variable selection, enhancing interpretability.
  • Dimension Reduction: Principal Component Regression (PCR) and Partial Least Squares (PLS) project the asset space onto a smaller set of latent factors before regression.
  • Factor Integration: The model can incorporate external market factors (e.g., Fama-French). Unlike traditional models that assume conditional independence, this method adaptively determines dependencies from the data.

Experimental Validation

The methods were evaluated across three distinct scenarios:

  1. Synthetic Data: Testing recovery of "ground truth" under Multivariate Normal and heavy-tailed Multivariate t-distributions.
  2. Historical Stock Returns: A portfolio test using monthly returns for 250 stocks from the NYSE and AMEX (1968–1998).
  3. High-Dimensional Scalability: Weekly returns for 2,461 stocks from the Russell 3000 Index (1973–2007) to test performance with thousands of assets.

Key Insights and Results

The study reveals that utilizing all available data, even under a normality assumption, consistently outperforms using only complete cases with more complex distributions. Principal Component Regression (PCR) emerged as a particularly robust workhorse in synthetic tests. Meanwhile, the Lasso provided valuable interpretability by setting negligible covariances to zero, effectively clustering the market.

In empirical backtests, Lasso-based estimators achieved the highest Sharpe ratios and lowest tracking errors relative to the S&P 500. While OLS methods failed when $p > n$, the parsimonious alternatives remained stable and produced positive-definite matrices. The authors suggest a "rule of thumb" ($p=0.25$) for when to switch from OLS to parsimonious methods, balancing computational speed with statistical stability. On the Russell 3000 dataset, the algorithm efficiently identified that 36% of asset pairs were marginally uncorrelated and 6% were conditionally independent.

Conclusion and Future Directions

The monomvn framework provides a scalable solution for high-dimensional covariance estimation with missing data. While it assumes data is Missing Completely at Random (MCAR) and follows a multivariate normal distribution, it proves remarkably robust in practice. Future research may explore the use of Copulas or Student-t distributions to better capture tail risks, or develop fully Bayesian implementations to quantify estimation risk more comprehensively.

Summary updated

Abstract

Quantitative portfolio allocation requires the accurate and tractable estimation of covariances between a large number of assets, whose histories can greatly vary in length. Such data are said to follow a monotone missingness pattern, under which the likelihood has a convenient factorization. Upon further assuming that asset returns are multivariate normally distributed, with histories at least as long as the total asset count, maximum likelihood (ML) estimates are easily obtained by performing repeated ordinary least squares (OLS) regressions, one for each asset. Things get more interesting when there are more assets than historical returns. OLS becomes unstable due to rank--deficient design matrices, which is called a "big p small n" problem. We explore remedies that involve making a change of basis, as in principal components or partial least squares regression, or by applying shrinkage methods like ridge regression or the lasso. This enables the estimation of covariances between large sets of assets with histories of essentially arbitrary length, and offers improvements in accuracy and interpretation. We further extend the method by showing how external factors can be incorporated. This allows for the adaptive use of factors without the restrictive assumptions common in factor models. Our methods are demonstrated on randomly generated data, and then benchmarked by the performance of balanced portfolios using real historical financial returns. An accompanying R package called monomvn, containing code implementing the estimators described herein, has been made freely available on CRAN.
BibTeX
@misc{gramacy2007estimating,
  author = {Robert B. Gramacy and Joo Hee Lee and Ricardo Silva},
  title = {{On estimating covariances between many assets with histories of highly variable length}},
  year = {2007},
  eprint = {0710.5837},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/0710.5837},
  howpublished = {arXiv},
}
Figure