Winning Models for GPA, Grit, and Layoff in the Fragile Families Challenge

Daniel E. Rigobon, Eaman Jahani, Yoshihiko Suhara, Khaled AlGhoneim, Abdulaziz Alghunaim, Alex Pentland, Abdullah AlmaatouqarXiv 2018 · DOI · arXiv

Abstract: In this paper, we discuss and analyze our approach to the Fragile Families Challenge. The challenge involved predicting six outcomes for 4,242 children from disadvantaged families from around the United States. The data consisted of over 12,000 features (covariates) about the children and their parents, schools, and overall environments from birth to age 9. Our approach relied primarily on existing data science techniques, including: (1) data preprocessing: elimination of low variance features, imputation of missing data, and construction of composite features; (2) feature selection through univariate Mutual Information and extraction of non-zero LASSO coefficients; (3) three machine learning models: Random Forest, Elastic Net, and Gradient-Boosted Trees; and finally (4) prediction aggregation according to performance. The top-performing submissions produced winning out-of-sample predictions for three outcomes: GPA, grit, and layoff. However, predictions were at most 20% better than a baseline that predicted the mean value of the training data of each outcome.

Introduction

Predicting the trajectory of a human life is notoriously difficult. The Fragile Families Challenge (FFC) tasked participants with predicting six key life outcomes—GPA, grit, material hardship, eviction, layoff, and job training—for children in disadvantaged households.

The primary technical hurdle was the "curse of dimensionality." The dataset provided over 12,000 features (covariates) describing the children’s environments from birth to age nine, but only 2,121 households had known training outcomes. This "p > n" problem (more features than observations) makes standard statistical models prone to extreme overfitting, requiring a highly disciplined approach to feature selection and model validation.

Methods

The team employed a modular pipeline consisting of four main stages to handle the high-dimensional social data.

Overview of the feature engineering and selection pipeline.
  1. Preprocessing & Engineering: The initial 12,942 features were reduced to 5,168 by removing low-variance and high-missingness data. One-hot encoding for categorical variables ballooned the feature set back up to ~24,000, which was supplemented by "composite features" based on social science literature regarding homelessness.
  2. Feature Selection: To make the data manageable, two distinct filters were used: Univariate Mutual Information (selecting the top K features) and LASSO (L1 Regularization) to force non-predictive coefficients to zero.
  3. Model Development: Three types of models were trained in parallel: Elastic Net (best for continuous outcomes like GPA), Random Forest (non-linear tree-based ensemble), and XGBoost (Gradient-Boosted Trees).
  4. Ensembling: The final submissions were a weighted team average, where weights were assigned based on individual model performance on the public leaderboard.

Datasets

The study utilized the Fragile Families and Child Wellbeing Study (FFCWS) dataset, which followed 4,242 American households for 15 years. The raw data included ~12,942 variables covering birth to age 9 (Wave 1 through Wave 4). The training set consisted of 2,121 households with provided outcomes, while evaluation was conducted on a "Leaderboard" set and a "Holdout" set of equal size.

Key Insights and Results

The MIT team’s approach was highly successful in the context of the competition, ranking 1st in GPA, grit, and layoff.

  • The Limits of Prediction: Despite winning, the results reveal a sobering reality. The winning GPA model was less than 20% better than a baseline that simply guessed the mean GPA, and the grit model showed less than 10% improvement. This suggests that social outcomes are inherently "noisy" and may have a fundamental ceiling of predictability.
  • Predictive Anchors: Feature importance analysis showed that the most predictive information consistently came from the mother’s survey responses and the most recent data collection point (Wave 5, age 9).
  • Feature Reduction: The pipeline successfully winnowed 24,000+ features down to as few as 65–527 (depending on the outcome) using LASSO while maintaining predictive power.

Limitations and Future Work

The authors identify several roadblocks for future social science machine learning. Standard regression models are not ideal for outcomes like GPA or grit, which have strict upper and lower bounds; they suggest integrating Tobit regression or other bounded models into standard ML libraries. Furthermore, the de-identification of data removed crucial community-level information, such as residential segregation, which is known to affect child well-being. Finally, they propose the creation of a queryable database of social science findings to help engineers automatically weight features that are known to be causally relevant, bridging the gap between ML expertise and domain knowledge.

Summary updated

Abstract

In this paper, we discuss and analyze our approach to the Fragile Families Challenge. The challenge involved predicting six outcomes for 4,242 children from disadvantaged families from around the United States. The data consisted of over 12,000 features (covariates) about the children and their parents, schools, and overall environments from birth to age 9. Our approach relied primarily on existing data science techniques, including: (1) data preprocessing: elimination of low variance features, imputation of missing data, and construction of composite features; (2) feature selection through univariate Mutual Information and extraction of non-zero LASSO coefficients; (3) three machine learning models: Random Forest, Elastic Net, and Gradient-Boosted Trees; and finally (4) prediction aggregation according to performance. The top-performing submissions produced winning out-of-sample predictions for three outcomes: GPA, grit, and layoff. However, predictions were at most 20% better than a baseline that predicted the mean value of the training data of each outcome.
BibTeX
@misc{rigobon2018winning,
  author = {Daniel E. Rigobon and Eaman Jahani and Yoshihiko Suhara and Khaled AlGhoneim and Abdulaziz Alghunaim and Alex Pentland and Abdullah Almaatouq},
  title = {{Winning Models for GPA, Grit, and Layoff in the Fragile Families Challenge}},
  year = {2018},
  eprint = {1805.11557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.11557},
}
Figure