Abstract
In this paper, we address the accuracy of the results for the overdetermined full rank linear least squares problem. We recall theoretical results obtained in Arioli, Baboulin and Gratton, SIMAX 29(2):413--433, 2007, on conditioning of the least squares solution and the components of the solution when the matrix perturbations are measured in Frobenius or spectral norms. Then we define computable estimates for these condition numbers and we interpret them in terms of statistical quantities. In particular, we show that, in the classical linear statistical model, the ratio of the variance of one component of the solution by the variance of the right-hand side is exactly the condition number of this solution component when perturbations on the right-hand side are considered. We also provide fragment codes using LAPACK routines to compute the variance-covariance matrix and the least squares conditioning and we give the corresponding computational cost. Finally we present a small historical numerical example that was used by Laplace in Theorie Analytique des Probabilites, 1820, for computing the mass of Jupiter and experiments from the space industry with real physical data.
Introduction
In many engineering and scientific fields, the Linear Least Squares Problem (LLSP) is the standard tool for estimating parameters from noisy data, typically formulated as $\min \|Ax - b\|_2$. Traditionally, researchers assess the sensitivity of the solution $x$ by examining the "condition number" of the matrix $A$. This global metric indicates how much errors in the data $b$ or the model $A$ might be amplified in the final result.
However, a single global condition number is often a blunt instrument. In complex models involving multiple parameters, some components of the solution may be remarkably stable while others are incredibly sensitive to tiny perturbations. For instance, when calculating the masses of multiple planets simultaneously, a researcher needs to know if the "Jupiter" parameter is reliable even if the "Uranus" parameter is numerically shaky. This paper bridges the gap between numerical linear algebra and statistics by providing a framework for component-wise conditioning—a method to measure the reliability of each individual part of the solution.
Bridging Numerical Stability and Statistics
The authors build upon the theoretical concept of "Partial Condition Numbers" (PCN), focusing on making these values both computable and interpretable for practitioners. A key contribution of this work is the "Aha" moment: the realization that numerical stability and statistical uncertainty are two sides of the same coin.
In a classical linear statistical model, the authors prove that the ratio of the variance of a specific solution component to the variance of the right-hand side noise is exactly the condition number of that component (assuming perturbations only on the right-hand side). In other words, what a computer scientist calls a "badly conditioned" component is exactly what a statistician identifies as a "high variance" estimate.
Algorithmic Efficiency and Implementation
To make these theoretical insights actionable, the paper details how to derive condition numbers using standard matrix factorizations. If a researcher has already solved the LLSP using a QR factorization—the standard numerical approach—the sensitivity of any single component can be computed with just a few triangular solves, requiring only $O(n^2)$ additional floating-point operations (flops). Computing the conditioning for all components costs approximately $O(n^3)$ flops, which is comparable to the cost of the initial solve, making it an affordable sanity check for large-scale parameter estimation.
To facilitate adoption, the authors provide concrete Fortran code snippets utilizing LAPACK routines such as DTRSV, DPOTRI, and DTRCON. By providing equivalents for ScaLAPACK, they ensure these methods scale to modern parallel supercomputers handling massive datasets.
Case Studies: From Laplace to Modern Geodesy
The authors validate their approach using two distinct datasets that span two centuries of scientific inquiry:
1. Historical Planetary Masses
The team analyzed a 19th-century dataset from the French astronomer Bouvart, which Pierre-Simon Laplace used in 1820 to calculate the masses of Jupiter, Saturn, and Uranus. This problem involved 129 observations and 6 variables. The component-wise analysis confirms Laplace’s own intuition: the mass of Jupiter was extremely well-conditioned and reliable, whereas the mass of Uranus was poorly conditioned. These findings align with the actual accuracy of Laplace's results when compared to modern NASA values.
2. Modern Satellite Geodesy
The methods were also applied to real-world physical data from the French Space Agency (CNES) and the GRACE satellite mission. This much larger problem involved roughly 166,000 observations to estimate 2,597 harmonic coefficients of Earth's gravity field. In this context, condition numbers varied wildly—from $10^2$ to $10^8$. A global condition number would have suggested the entire solution was moderately risky, masking the fact that many components were perfectly stable while others were nearly useless. The paper also demonstrates how regularization techniques (such as the Kaula technique) specifically "tame" these high-degree components, bringing their condition numbers down to manageable levels.
Conclusion
This work provides a practical framework for determining the reliability of individual parameters in linear least squares problems. By linking numerical conditioning to statistical variance and providing efficient algorithms for their computation, the authors offer a vital tool for scientists and engineers. While the current study focuses on full-rank linear problems, it lays the groundwork for future explorations into rank-deficient cases and the conditioning of iterative steps in non-linear estimation.
Summary updated