Learning Measurement Models for Unobserved Variables

Ricardo Bezerra de Andrade e Silva, Richard Scheines, Clark Glymour, Peter SpirtesUAI 2003 · OpenReview

TL;DR

This paper introduces a novel algorithm for discovering latent variables and their causal relationships from observed data, even when the underlying structure is complex and unobserved. It addresses a fundamental challenge in science and data analysis: inferring hidden causes from indirect measurements, offering a principled approach that is asymptotically correct under standard assumptions and doesn't require prior knowledge of the number of latent variables.

What Problem Does This Paper Address?

Many scientific endeavors aim to understand underlying causal mechanisms or compositions (e.g., what causes a disease, what minerals make up soil). However, we rarely measure these scientifically important "latent" variables directly. Instead, we rely on indirect measurements of their effects. This gap between observed data and the underlying causal reality poses a significant challenge for automated data analysis. Existing methods like Principal Components Analysis or Factor Analysis often make strong, sometimes arbitrary, assumptions or can be too complex to interpret causally. While Bayesian networks offer a powerful framework for causal discovery, applying them to latent variable models is notoriously difficult due to irregular likelihood surfaces and ill-defined model dimensions. This paper tackles the problem of learning the structure of these latent variables and their relationships with observed indicators, a crucial step for building more accurate and interpretable causal models.

Methods

The core contribution is an algorithm designed to learn "measurement patterns," which represent equivalence classes of latent variable models. The algorithm operates under the assumption of a "purifiable linear latent variable graph," meaning observed variables are linear functions of their latent causes, and error terms are additive. Key aspects of the method include:

  • Tetrad Constraints: The algorithm leverages tetrad constraints (relationships among four variables) to identify potential latent structures. Specific patterns of vanishing tetrads are characteristic of different underlying causal structures, particularly those involving a single latent cause for a set of observed variables.
  • Clustering and Impurity Identification: The algorithm iteratively clusters observed variables that likely share a common latent cause. It employs an "Unclustered" test to identify pairs of variables that cannot be explained by a simple one-factor model, helping to refine these clusters and identify "impurities" (where observed variables might be influenced by multiple latents or have correlated errors).
  • Measurement Pattern Output: Instead of a single graph, the algorithm outputs a "measurement pattern," a graphical object that captures an equivalence class of indistinguishable measurement models. This acknowledges that, from observed data alone, multiple latent structures might be statistically equivalent.
  • Purification: The measurement pattern can then be "purified" to yield pure measurement models, where each observed variable has a single latent parent.

Datasets

The paper primarily uses simulated datasets for evaluation. These simulations are designed to represent various scenarios, including:

  • Linear relations among latents with pure measurement models.
  • Linear relations among latents with impure measurement models.
  • Non-linear relations among latents with impure measurement models.

The authors also mention a real-world example involving indicators of "student anxiety" that was analyzed using their method. No new benchmarks are introduced; rather, existing simulation paradigms are employed to test the algorithm's performance.

Key Insights and Ideas

  • Tetrads as Building Blocks: The paper demonstrates the power of tetrad constraints, a concept from causal discovery, as fundamental building blocks for inferring latent structures. This provides a principled way to move beyond simple correlations.
  • Handling Impurity is Crucial: The algorithm's ability to identify and handle "impure" measurement models (where observed variables might have multiple latent causes or correlated errors) is a significant advantage over simpler methods. This makes it more robust to real-world data complexities.
  • Equivalence Classes are Necessary: Recognizing that latent variable models are often not uniquely identifiable from observed data, the output of a "measurement pattern" (representing an equivalence class) is a more accurate and honest representation of what can be learned.
  • Number of Indicators Matters: The empirical evaluation highlights that the number of indicators per latent variable is a critical factor for successful discovery. Having at least four indicators per latent significantly improves performance, suggesting that redundancy in measurement is key to robust inference.

Key Results

The algorithm's performance is evaluated through several simulation studies. Key findings include:

  • High Accuracy with Sufficient Indicators: When latents have four or more indicators, the algorithm achieves very good results, with low proportions of missing latents, missing indicators, and misplaced measurements. Increasing sample size had diminishing returns once a sufficient number of indicators was present.
  • Robustness to Impurity: Study 2, which introduced impure indicators, showed largely unchanged results, indicating the algorithm's robustness to measurement model impurity.
  • Performance on Non-linear Models: Study 3, using a non-linear latent structure, showed reasonable results, though the paper notes that the Wishart test (used for tetrad constraints) assumes multivariate normality, and further investigation with non-Gaussian data is warranted.
  • Comparison to Factor Analysis: The paper briefly compares its approach to Factor Analysis, noting that SAS's default criterion for choosing the number of latents often underestimated the true number, while their method does not require this prior specification.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Computational Complexity: The algorithm's worst-case complexity is exponential due to clique finding. While it performs well in practice for less impure graphs, scaling to very large problems remains a challenge. The authors suggest exploring approximate algorithms.
  • Linearity Assumption: The core algorithm assumes linear relationships between observed variables and their latent causes. Future work could explore extensions to discrete variables and potentially non-linear latent relationships.
  • Further Empirical Evaluation: More extensive experimental evaluation is needed, including tests with non-Gaussian data and real-world datasets where assumptions might not fully hold.
  • Bayesian/Score-Based Methods: While their previous attempts with Bayesian methods were not as successful, exploring these approaches further for latent variable models is a promising avenue.

Summary updated

Abstract

Observed associations in a database may be due in whole or part to variations in unrecorded (latent) variables. Identifying such variables and their causal relationships with one another is a principal goal in many scientific and practical domains. Previous work shows that, given a partition of observed variables such that members of a class share only a single latent common cause, standard search algorithms for causal Bayes nets can infer structural relations between latent variables. We introduce an algorithm for discovering such partitions when they exist. Uniquely among available procedures, the algorithm is (asymptotically) correct under standard assumptions in causal Bayes net search algorithms, requires no prior knowledge of the number of latent variables, and does not depend on the mathematical form of the relationships among the latent variables. We evaluate the algorithm on a variety of simulated data sets.
BibTeX
@inproceedings{silva2003learning,
  author = {Ricardo Bezerra de Andrade e Silva and Richard Scheines and Clark Glymour and Peter Spirtes},
  title = {{Learning Measurement Models for Unobserved Variables}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2003},
  url = {https://openreview.net/forum?id=O5Bz0RLAxWd},
}
Figure