TL;DR
This paper introduces novel machine learning techniques, specifically recursive partitioning inspired by random forests, to detect Differential Item Functioning (DIF) in complex survey data. These methods efficiently identify subgroups within a population that exhibit biased item responses, even when the underlying measurement model is multidimensional and uses ordinal data, offering a data-driven alternative to traditional, theory-driven approaches.
What Problem Does This Paper Address?
In social science surveys, it's crucial that the measurement of latent traits (like creativity or anxiety) is consistent across different groups of people. This is known as measurement invariance. When this invariance is violated, it's called Differential Item Functioning (DIF), meaning an item behaves differently for individuals with the same underlying trait but different group memberships (e.g., based on demographics). Detecting DIF is vital because undetected DIF can lead to misinterpretations of group differences in latent traits. Traditional DIF detection methods often require researchers to pre-specify the groups to examine, which can miss subtle DIF occurring in less obvious subgroups. This paper addresses the challenge of identifying these unknown subgroups exhibiting DIF, particularly in complex survey data characterized by multiple latent variables and ordinal response scales (like Likert scales).
Methods
The core contribution of this paper lies in adapting and extending machine learning techniques, specifically model-based recursive partitioning (MOB), for DIF detection in multidimensional graded response (MGR) models.
- Recursive Partitioning (Tree-Based Methods): The authors leverage tree-based algorithms to recursively split the sample into subgroups based on partitioning variables (covariates). At each split, they test for "parameter instability," indicating that item parameters differ across potential subgroups.
- Model-Based Recursive Partitioning (MOB): This approach integrates statistical model estimation (MGR models) with the recursive partitioning process. The goal is to find subgroups where the MGR model fits well but exhibits DIF.
- Extensions to MGR Models: The paper adapts existing methods like PCM Tree and SEM Trees to handle the complexities of MGR models, which involve multiple latent variables and ordinal response data. They highlight that direct application of some methods is computationally infeasible for MGR models.
- Limited Information Estimation: To manage computational costs, they propose using limited information estimation (like ML or WLS) during the tree-growing process, which is more efficient than full information estimation.
- Random Forest-Inspired Extension: To address the instability of single decision trees (where small data changes can drastically alter the tree structure), they propose a robust, scalable extension inspired by random forests. This involves growing multiple trees with random variable selection at each split and aggregating their results. This ensemble approach helps to identify stable subgroups and mitigate the impact of noise.
Datasets
The paper uses simulated data for evaluation.
- Simulation 1: Data was generated based on a probit multistate IRT model with latent item effect variables for graded responses (PIEG), mimicking a longitudinal setting with ordinal response variables. A traditional CFA model was also used to generate numerical data for comparison.
- Simulation 2: Data was generated to create more complex subgroup structures where the MGR model might not hold for all individuals, and the subgroups are not easily separable by a single decision tree.
Key Insights and Ideas
- Data-Driven Subgroup Discovery: The primary insight is that machine learning, particularly recursive partitioning, offers a powerful data-driven way to discover unknown subgroups exhibiting DIF, bypassing the need for strong theoretical priors about group membership.
- Handling Multidimensionality and Ordinal Data: The proposed methods are specifically tailored to address the complexities of multidimensional latent variables and ordinal response data, which are common in social science surveys but challenging for traditional DIF detection.
- Random Forests for Robustness: The random forest-inspired extension is crucial. It highlights that single decision trees can be unstable. Ensemble methods (forests) provide a more robust way to identify DIF subgroups by aggregating results from multiple, decorrelated trees, making the findings more reliable.
- Computational Feasibility: The paper emphasizes the importance of computational efficiency. By using limited information estimation during tree growing, they make MOB for MGR models practical for larger datasets.
- Interpretable Decision Rules: A significant advantage of tree-based methods is that the identified subgroups are defined by clear, interpretable decision rules (e.g., "if num1 > 100 and cat1 is in {2,3,4}").
Key Results
- Successful Subgroup Detection: In simulation 1, both the partykit and score-guided semtree algorithms (single trees) correctly identified the simulated subgroups with DIF, especially when the data clearly reflected the subgroup structure.
- Ensemble Performance: Simulation 2 demonstrated that recursive partitioning forests (using partykit and semtree) are effective in retrieving DIF subgroups even in more complex scenarios where a single tree struggles.
- Computational Efficiency: Score-guided semtree was found to be significantly faster than naive semtree and partykit for single tree growing. Forest computations were also manageable, especially with parallelization.
- Tree Stability: While single trees can be unstable, ensemble methods (forests) showed improved stability and robustness in identifying subgroups across multiple simulations. Numerical data tended to yield more perfectly stable trees in simulation 1, but this was attributed to split point inaccuracies rather than fundamentally better performance.
- Model Fit as a Criterion: The methods not only identify potential DIF subgroups but also provide model fit indices (like RMSEA) for these subgroups, allowing researchers to select subgroups where the assumed MGR model holds well.
Limitations and Future Work
- Computational Cost of Full Information: The paper notes that using full information estimation for MGR models within recursive partitioning is computationally infeasible due to high costs.
- Assumptions in Tree Growing: Using limited information estimation during tree growing might lead to more inaccurate split points, particularly for numerical partitioning variables, when MGR model assumptions are compromised.
- Interpretation of Subgroups: The authors caution that not all subgroups identified with acceptable model fit indices in tree ensembles should be strictly interpreted as true DIF subgroups; those with the best and unique model fit are more reliable.
- Future Work: The paper suggests further extensions could include exploring bootstrap resampling in tree ensembles for MGR models to enhance robustness. They also acknowledge that while their methods are designed for ordinal data, they rely on assumptions of continuous multivariate distributions during tree growing, which could be a point of further investigation.
Summary updated