Latent Variable Forests for Latent Variable Score Estimation

Franz Classe, Christoph KernEducational 2024 · DOI

TL;DR

This paper introduces "Latent Variable Forests" (LV Forest), a novel machine learning approach to estimate latent variable scores more accurately, especially when underlying model parameters vary across different subgroups of the population. By combining confirmatory factor analysis with tree-based methods, LV Forest can identify these subgroups and provide more reliable and interpretable scores, addressing a key challenge in psychological and educational measurement.

What Problem Does This Paper Address?

In many fields, particularly psychology and education, researchers use questionnaires to measure unobservable "latent variables" (e.g., intelligence, personality, life satisfaction). Estimating scores for these latent variables is crucial for diagnosis, assessment, and understanding individual differences. A significant challenge arises when the relationship between observed responses (items in a questionnaire) and the latent variable isn't consistent across all individuals. This inconsistency, known as "parameter heterogeneity," can occur due to "differential item functioning" (DIF) or other "construct-irrelevant" variables (like gender, ethnicity, or socioeconomic status). If not accounted for, this heterogeneity can lead to biased and inaccurate latent variable scores, misinterpreting group differences as bias rather than genuine variations in the latent trait. Traditional methods often require a priori specification of these subgroups, which is not always feasible.

Methods

The core innovation is the Latent Variable Forest (LV Forest) algorithm. It's a hybrid approach that leverages:

  • Confirmatory Factor Analysis (CFA) / Item Response Theory (IRT): These are standard statistical models used to represent the relationship between observed item responses and latent variables. LV Forest uses these models to estimate parameters and scores.
  • Tree-Based Machine Learning (Specifically, SEM Trees): This is the novel component. LV Forest uses a decision tree-like structure to recursively partition the data into subgroups. At each split, it tests for parameter instability (heterogeneity) using a generalized M-fluctuation test. If significant instability is found, the data is split based on a covariate that best explains this instability.
  • Ensemble Learning (Random Forests Inspiration): To ensure robustness against small data fluctuations, LV Forest builds an ensemble of multiple decision trees. This involves random selection of partitioning variables and bootstrap aggregating (bagging) to create diverse trees.
  • Pruning and Model Fit Criteria: Only terminal nodes (subgroups) that demonstrate good model fit (e.g., low RMSEA) and stable parameters are used for score estimation. This ensures that the estimated scores are based on reliable models.
  • Empirical Bayes Modal (EBM) Estimation: Latent variable scores are then computed within these selected subgroups using EBM, which combines model parameters with prior distributions for robust estimation.

Datasets

The paper uses two main types of data for evaluation:

  • Simulated Data: The authors generated data based on a univariate Graded Response Model (GRM) with varying parameter heterogeneity across simulated subgroups. This allowed them to control the data-generating process and evaluate the performance of LV Forest under known conditions.
  • Real-World Survey Data: They utilized data from the Longitudinal Internet studies for the Social Sciences (LISS) panel, specifically focusing on a life satisfaction scale. This dataset included several background variables (gender, income, education, etc.) used as partitioning variables to identify relevant subgroups.

Key Insights and Ideas

  • Automated Subgroup Detection: LV Forest can automatically identify relevant subgroups where parameter heterogeneity exists, removing the need for a priori hypotheses about these groups.
  • Unbiased and Interpretable Scores: By estimating scores within homogeneous subgroups, LV Forest provides scores that are interpretable with respect to subgroup differences without conflating these differences with measurement bias.
  • Robustness through Ensembles: Using an ensemble of trees makes the method less sensitive to the specific partitioning variables or data splits, leading to more stable results.
  • Parameter Heterogeneity is Key: The method's advantage is most pronounced when parameter heterogeneity is present. In the absence of such heterogeneity, LV Forest essentially reverts to simpler models.

Key Results

  • Improved Accuracy: In simulations with parameter heterogeneity, LV Forest significantly improved the accuracy of latent variable score estimation compared to naive models (fitting a single model to all data) and single SEMTrees. Correlations with true scores were notably higher.
  • Reduced Nonconvergence: While LV Forest can have higher nonconvergence rates than naive models, it generally performs better than single SEMTrees in terms of providing estimates for a larger proportion of individuals, especially in complex subgroup structures.
  • Real-World Application: In the LISS data analysis, LV Forest identified several relevant subgroups and provided latent variable scores that were largely independent of construct-irrelevant variables within those subgroups, suggesting interpretable subgroup differences rather than bias.
  • Computational Efficiency: While building an ensemble of trees is more computationally intensive than a single tree, the method is designed to be efficient, especially with parallel processing.

Limitations and Future Work

  • Sample Size Requirements: LV Forest requires sufficiently large sample sizes, particularly in the terminal nodes of the decision trees, to reliably estimate model parameters and perform M-fluctuation tests.
  • Measurement of Construct-Irrelevant Variables: The method's effectiveness relies on measuring the relevant construct-irrelevant variables that cause parameter heterogeneity. If these are not measured, it can lead to nonconvergence for some individuals.
  • Interpretability of Subgroups: While LV Forest identifies subgroups, the specific composition of these subgroups might not always be directly interpretable in a generalizable way, as the focus is on score estimation.
  • M-Fluctuation Test for Ordinal Data: While the paper argues for its reliability with ordinal data, the reliance on ML estimation for the M-fluctuation test can be a concern if the normality assumption is severely violated.
  • Future Work: The authors suggest that LV Forest is a valuable tool for complex SEMs and for estimating effects between latent variables. Further research could explore its application in different modeling contexts and with various types of response variables.

Summary updated

Abstract

We develop a <i>latent variable forest</i> (LV Forest) algorithm for the estimation of latent variable scores with one or more latent variables. LV Forest estimates unbiased latent variable scores based on <i>confirmatory factor analysis</i> (CFA) models with ordinal and/or numerical response variables. Through parametric model restrictions paired with a nonparametric tree-based machine learning approach, LV Forest estimates latent variable scores using models that are unbiased with respect to relevant subgroups in the population. This way, estimated latent variable scores are interpretable with respect to systematic influences of covariates without being biased by these variables. By building a tree ensemble, LV Forest takes parameter heterogeneity in latent variable modeling into account to capture subgroups with both good model fit and stable parameter estimates. We apply LV Forest to simulated data with heterogeneous model parameters as well as to real large-scale survey data. We show that LV Forest improves the accuracy of score estimation if parameter heterogeneity is present.
BibTeX
@inproceedings{classe2024latent,
  author = {Franz Classe and Christoph Kern},
  title = {{Latent Variable Forests for Latent Variable Score Estimation}},
  booktitle = {Educational},
  year = {2024},
  doi = {10.1177/00131644241237502},
}
Figure