TL;DR
This paper introduces Copula-EM, a novel semiparametric algorithm for imputing missing values in mixed-type datasets (continuous, Boolean, and ordinal). By modeling the data's joint distribution using a Gaussian copula, Copula-EM effectively captures complex dependencies and outperforms existing methods in both accuracy and speed, particularly for datasets with many ordinal variables.
What Problem Does This Paper Address?
Missing data is a pervasive problem in data analysis, and imputing it accurately is crucial for subsequent analyses. Standard imputation techniques often struggle with "mixed data" – datasets containing a combination of continuous, Boolean (binary), and ordinal variables. These methods can fail basic sanity checks, like imputing values that don't align with the original data's distributions. For instance, treating ordinal data (like survey responses on a Likert scale) as continuous can lead to nonsensical results. The challenge lies in modeling the intricate relationships between these different data types, especially when dealing with many ordinal categories or high missingness rates.
Methods
The core of the proposed solution is a Gaussian copula model. This statistical framework allows for modeling the joint distribution of variables by separating the marginal distributions of individual variables from their dependence structure. Specifically, the authors assume that each variable in the mixed dataset can be represented by a latent, underlying continuous variable that follows a standard normal distribution. For continuous variables, the observed value is a direct transformation of its latent counterpart. For ordinal variables, the observed value is determined by thresholding the latent variable. Boolean variables are treated as a special case of ordinal variables with two levels.
To estimate the parameters of this Gaussian copula model from incomplete mixed data, the authors develop an efficient approximate Expectation-Maximization (EM) algorithm. The EM algorithm iteratively refines estimates of the model parameters. The "E-step" involves computing the expected values of the latent variables given the observed data and current parameter estimates, while the "M-step" updates the model parameters to maximize the expected log-likelihood. The "approximate" nature of their EM algorithm, particularly in handling truncated normal distributions, is key to its speed advantage.
Datasets
The paper evaluates its method on several datasets:
- Synthetic Data: Generated from a specified Gaussian copula model with varying data types (exponential for continuous, binary, and 1-5 ordinal) and missingness rates (30%).
- General Social Survey (GSS) Data: A real-world dataset with 17 ordinal variables (2-48 levels) and 1 continuous variable, featuring a natural missingness rate of 24.9%.
- MovieLens 1M Data: A subset of a popular movie recommendation dataset, characterized by a very high missingness rate (75.6%).
- CAL500exp Data: An enriched music auto-tagging dataset with 67 binary tags.
- Additional Ordinal and Mixed Datasets: Including Lecturers Evaluation (LEV), Employee Selection (ESL), German Breast Cancer Study Group (GBSG), and Restaurant Tips (TIPS).
Key Insights and Ideas
- Gaussian Copula as a Unifying Framework: The Gaussian copula elegantly handles mixed data by treating all variables as transformations of latent normal variables, allowing for a unified approach to imputation.
- Separation of Marginal and Dependence Structures: A key advantage of copulas is their ability to model dependence separately from marginal distributions. This means the imputation accuracy is less sensitive to the exact specification of marginal distributions, as long as they are handled correctly (e.g., via thresholding for ordinals).
- Approximate EM for Speed: The development of an approximate EM algorithm is crucial. While exact EM for this problem is computationally prohibitive, their approximation significantly speeds up convergence without a substantial loss in accuracy, making it practical for larger datasets.
- Interpretability of Latent Structure: The fitted copula model can reveal statistical associations among variables, offering insights beyond just imputation.
Key Results
- Superior Imputation Accuracy: Copula-EM consistently outperforms state-of-the-art nonparametric imputation algorithms (missForest, xPCA, imputeFAMD) and low-rank matrix completion methods (GLRM, softImpute) across various synthetic and real-world datasets, especially for ordinal and mixed data. For example, on synthetic data, it substantially outperforms nonparametric algorithms for all data types. On GSS data, it achieves the best SMAE for selected variables.
- Significant Speedup: Copula-EM is substantially faster than existing methods, particularly the MCMC-based Gaussian copula estimator (sbgcop). For instance, on synthetic data, Copula-EM converges in about 25s, while sbgcop takes much longer and achieves lower accuracy at shorter times. Even when sbgcop is given more runtime, Copula-EM often estimates correlations and continuous imputations more accurately.
- Robustness to Missingness: The method shows strong performance even with high missingness ratios (e.g., 75.6% in MovieLens 1M).
- No Tuning Parameters: A significant practical advantage is that Copula-EM requires no hyperparameter tuning, simplifying its application.
Limitations and Future Work
- Scalability to High Dimensions: While faster than many alternatives, the computational complexity of Copula-EM scales cubically with the number of variables (O(anp³)). Scaling to datasets with a very large number of features remains an area for future work.
- Missing Completely At Random (MCAR) Assumption: The estimation of marginal functions relies on the MCAR assumption. While the paper notes this may be relaxed to MAR or even missing not at random by carefully modeling the missing mechanism, this is left for future research.
- Binary Data Performance: The paper notes a slight drop in imputation accuracy for binary data compared to other methods, which they conjecture might be mitigated by using multiple imputation.
- Further Exploration of Approximations: While the approximate EM is effective, further research into its impact on different data types and potential improvements is suggested.
Summary updated