TL;DR
This paper introduces CrowdMI, a novel framework that leverages crowdsourcing to perform multiple imputation of missing data. By transforming missing data points into survey questions, CrowdMI demonstrates that humans can impute both qualitative and quantitative data with accuracy comparable to complex statistical models, opening new avenues for data completion in various domains.
What Problem Does This Paper Address?
Missing data is a pervasive issue in data analysis, often leading to biased inference and reduced sample sizes. While statistical and machine learning models offer imputation techniques, they face limitations such as sample size requirements, sensitivity to missingness patterns, and computational complexity. The paper questions whether humans, with their intuitive reasoning and adaptability, can effectively perform multiple imputation, a process crucial for accounting for imputation uncertainty. This research aims to bridge the gap between human cognitive abilities and the need for robust missing data handling.
Methods
CrowdMI's core innovation lies in its approach to structuring the imputation task as a survey questionnaire. Missing data points are extracted from a dataset and presented to crowdworkers within a survey. This survey provides context about the dataset and poses specific questions to elicit imputed values. The process is designed to mimic multiple imputation by having multiple crowdworkers independently impute values for the same missing data point. The authors emphasize designing simple, visually appealing, and contextually rich questionnaires to maximize response quality. They also explore the impact of providing varying degrees of prior knowledge to crowdworkers.
Datasets
The paper evaluates CrowdMI using three datasets:
- FEV Dataset: Explores the relationship between respiratory function (Forced Expiratory Volume - FEV) and smoking.
- Pima Indian Diabetes Dataset: Contains information on diabetes status for females aged 21 and older.
- Perturbed FEV Dataset: A modified version of the FEV dataset used to test the robustness of CrowdMI to systematic changes in data.
Key Insights and Ideas
- Human Imputation is Viable: The study confirms that humans can effectively impute both qualitative and quantitative missing data, challenging the notion that only complex algorithms can handle this task.
- Crowdsourcing as a Multiple Imputation Framework: CrowdMI successfully replicates the core idea of multiple imputation by generating multiple, slightly varied imputations for each missing value through independent crowdworker contributions.
- Questionnaire Design is Crucial: The quality of crowdworker responses is highly dependent on the design of the survey. Simple language, visual aids (like scatter plots), and clear instructions significantly improve imputation accuracy and reduce noisy answers.
- Context Matters: Providing relevant contextual information about the dataset and attribute correlations is essential for crowdworkers to make informed imputations.
- Human vs. Machine Imputation: CrowdMI's performance is shown to be comparable to, and in some cases better than, a state-of-the-art machine imputation method (MICE), especially in terms of variability and coverage of original values.
Key Results
- Comparable Accuracy: CrowdMI's imputed values were found to be closer to the original values than those imputed by MICE in several instances, particularly for the FEV dataset.
- Reduced Variability: CrowdMI imputations exhibited less variability compared to MICE, suggesting a more consistent human judgment.
- Categorical Imputation Challenges: While CrowdMI performed well, imputing categorical variables (like gender) proved more challenging and error-prone than continuous variables, though still comparable to MICE.
- Impact of Visualizations: The inclusion of scatter plots significantly improved the quality of imputations, highlighting the importance of effective data visualization in crowdsourcing tasks.
- Participant Satisfaction: Crowdworkers generally found the surveys to be of "easy" to "average" difficulty, with a high satisfaction rate regarding compensation and clarity of instructions.
Limitations and Future Work
The authors acknowledge that their study used only three datasets, which might limit the generalizability of their findings. Future work should focus on validating CrowdMI on a wider variety of datasets, particularly large-dimensional ones where relationships between variables are less obvious. They also suggest exploring more complex scenarios where multiple variables are missing simultaneously and investigating how to further influence crowd imputation through strategic placement of contextual information within the questionnaire.
Summary updated