Finiteness of small factor analysis models

Mathias Drton, Xiao Han

TL;DR

This paper proves a "finiteness" property for factor analysis models with one or two factors. This means that for a given number of factors, there's a specific size of covariance matrix beyond which you can determine if a matrix belongs to the factor model by only examining smaller sub-matrices. This has implications for statistical inference in high-dimensional settings.

What Problem Does This Paper Address?

Factor analysis is a statistical technique used to explain the correlations among observed variables by assuming they are influenced by a smaller number of unobserved latent factors. A key question in factor analysis is whether a given covariance matrix can be represented by a specific factor model. This is often formulated as testing if a covariance matrix belongs to a particular set, denoted as $F_{p,m}$, where $p$ is the number of observed variables and $m$ is the number of factors.

The challenge arises when dealing with high-dimensional data (large $p$). Traditional methods for testing if a covariance matrix belongs to $F_{p,m}$ can become computationally intractable or statistically unreliable as $p$ grows. This paper addresses the question of whether there's a structural property that simplifies this determination in high dimensions. Specifically, they investigate a "finiteness" property: does the ability to determine if a covariance matrix belongs to $F_{p,m}$ depend only on a finite number of variables, regardless of how large $p$ becomes?

Methods

The core of the paper's methodology lies in algebraic and geometric arguments applied to the structure of the factor analysis model. The factor model defines the covariance matrix $\Sigma$ as $\Sigma = \Delta + \Gamma\Gamma^t$, where $\Delta$ is a positive definite diagonal matrix (representing unique variances) and $\Gamma$ is a $p \times m$ matrix of factor loadings.

The authors leverage several key mathematical tools:

  • Algebraic Statistics: They treat the set of valid covariance matrices for a factor model as an algebraic variety.
  • Principal Submatrices: The central idea is to relate the properties of the full $p \times p$ covariance matrix to the properties of its principal submatrices (submatrices formed by selecting the same rows and columns).
  • Induction: The proofs for the main theorems (for $m=1$ and $m=2$) heavily rely on induction, building up results from smaller matrix sizes to larger ones.
  • Linear Algebra: Techniques like matrix rank, determinants, and orthogonal transformations are used extensively to analyze the structure of $\Gamma$ and $\Delta$.
  • Lemmas for Uniqueness: Several lemmas are introduced to establish conditions under which the representation $\Sigma = \Delta + \Gamma\Gamma^t$ is unique, which is crucial for their finiteness arguments. For instance, Lemma 2 provides conditions for the uniqueness of diagonal entries based on the properties of submatrices.

The novelty lies in formalizing and proving this "finiteness" property, which implies that for a fixed number of factors $m$, there exists a critical size $2(m+1)$ for principal submatrices. If all principal submatrices of this size satisfy the conditions for belonging to $F_{2(m+1), m}$, then the larger matrix $\Sigma$ also belongs to $F_{p,m}$.

Datasets

Not discussed in the paper. The paper is theoretical and focuses on the mathematical properties of covariance matrices within the factor analysis model, rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Finiteness as a Structural Property: The most significant insight is that factor analysis models possess a "finiteness" property. This means that for a fixed number of factors ($m$), the condition for a covariance matrix to belong to the model space $F_{p,m}$ depends only on submatrices of a fixed size, $2(m+1)$, regardless of the total number of variables $p$.
  • Distinguished Matrix Size: The paper identifies this critical size as $2(m+1)$. For $m=1$ factor, this size is 4. For $m=2$ factors, it's 6. This implies that if all $4 \times 4$ principal submatrices of a covariance matrix belong to $F_{4,1}$, then the entire $p \times p$ matrix (for $p>4$) belongs to $F_{p,1}$.
  • Implications for High-Dimensional Inference: This finiteness property suggests that statistical tests for factor models in high dimensions might be simplified. Instead of analyzing the entire large covariance matrix, one could potentially focus on testing smaller, fixed-size principal submatrices. This could lead to more efficient and robust inferential procedures.
  • Algebraic Geometry Connection: The paper mentions connections to algebraic geometry, suggesting that these finiteness properties are deeply rooted in the geometric structure of the factor analysis model space.

Key Results

  • Theorem 1: For $m \in \{0, 1, 2\}$ factors, a $p \times p$ covariance matrix $\Sigma$ belongs to $F_{p,m}$ if and only if all its principal submatrices of size $2(m+1)$ belong to $F_{2(m+1), m}$, provided $p > 2(m+1)$.
  • One-Factor Model ($m=1$): A $p \times p$ matrix ($\Sigma \in F_{p,1}$) if and only if all $4 \times 4$ principal submatrices belong to $F_{4,1}$ and at least one $2 \times 2$ principal submatrix does not belong to $F_{2,0}$ (i.e., it's not diagonal). This implies that for $m=1$, the critical size is indeed 4.
  • Two-Factor Model ($m=2$): A $p \times p$ matrix ($\Sigma \in F_{p,2}$) if and only if all $6 \times 6$ principal submatrices belong to $F_{6,2}$ and at least one $4 \times 4$ principal submatrix does not belong to $F_{4,1}$. This establishes the critical size of 6 for $m=2$.

Limitations and Future Work

  • Limited to $m \le 2$: The main results are proven only for models with up to two factors ($m \le 2$). The authors note that their arguments become significantly more complex for $m \ge 3$ and are difficult to extend directly.
  • Distribution Theory Needed: While the paper proves the structural property of finiteness, it explicitly states that developing the appropriate distribution theory for statistical tests based on these findings is beyond the scope of this work. This is a crucial next step for practical applications.
  • Generalization to $m \ge 3$: The paper conjectures that the critical matrix size of $2(m+1)$ might hold for any number of factors $m$. However, proving this for $m \ge 3$ remains an open challenge.
  • Uniqueness of Representation: The paper explores conditions for the uniqueness of the $\Delta + \Gamma\Gamma^t$ representation, which is important for the finiteness proofs. However, for $m=2$, it shows that the representation is not always unique, leading to more intricate arguments.

The authors suggest that future work should focus on extending these results to higher numbers of factors and developing the statistical inference procedures that leverage this finiteness property.

Summary updated

Abstract

We consider small factor analysis models with one or two factors. Fixing the number of factors, we prove a finiteness result about the covariance matrix parameter space when the size of the covariance matrix increases. According to this result, there exists a distinguished matrix size starting at which one can determine whether a given covariance matrix belongs to the parameter space by determining whether all principal submatrices of the distinguished size belong to the corresponding parameter space. We show that the distinguished matrix size is equal to four in the one-factor model and six with two factors.
BibTeX
@article{drton2010finiteness,
  author = {Mathias Drton and Xiao Han},
  title = {{Finiteness of small factor analysis models}},
  journal = {Annals},
  year = {2010},
  doi = {10.1007/s10463-010-0293-6},
}
Figure