Scalable Early Childhood Reading Performance Prediction

Zhongkai Shangguan, Zanming Huang, Eshed Ohn-Bar, Ola Ozernov-Palchik, Derek Kosty, Michael Stoolmiller, Hank FienNeurIPS 2024

In the United States, roughly 37% of fourth graders do not read at grade level. Historically, schools have relied on a "wait-to-fail" approach—waiting for a child to fall significantly behind before intervening. While Machine Learning (ML) could theoretically identify at-risk students much earlier, the field has lacked large-scale, publicly available datasets to train these predictive models.

Furthermore, educational data is notoriously difficult to model. It is typically "tabular" (rows and columns of scores and demographics) and riddled with missing values due to student absences, school transfers, or incomplete documentation. Standard ML techniques often struggle with these gaps, either requiring noisy imputation (filling in the blanks) or discarding valuable partial data entirely.

To address this, researchers have introduced the ECRI dataset, a first-of-its-kind large-scale longitudinal benchmark for predicting early childhood reading outcomes, alongside a novel self-supervised pre-training strategy called MaskMLP.

The ECRI Dataset

The paper introduces the Enhanced Core Reading Instruction (ECRI) dataset to serve as a robust benchmark for educational AI:

  • Scale: 6,916 first-grade students and 172 teachers across 44 US schools.
  • Features: 16 input variables, including demographics (age, gender, socio-economic status) and eight initial literacy assessments (e.g., Woodcock Reading Mastery, DIBELS, Stanford Achievement Test).
  • The "Messiness" Factor: Approximately 30.48% of the assessment data is missing, accurately reflecting real-world educational environments.

The Core Innovation: MaskMLP

Instead of using standard imputation methods like filling missing values with zeros or means, the authors developed MaskMLP. This self-supervised pre-training strategy learns robust representations directly from messy tabular data.

Architecture of the MaskMLP pre-training strategy

The framework operates in four key steps:

  1. Masking: An input vector (student data) is taken, and 25% of the observed features are randomly hidden.
  2. Embedding Alignment: Both the original data and the masked data are passed through a Multi-Layer Perceptron (MLP) to generate latent embeddings.
  3. Self-Supervised Loss: The model is trained to minimize the cosine embedding loss between the original and masked versions. This forces the model to learn the underlying relationships between variables, allowing it to "infer" missing information in the latent space without explicitly filling in the raw data.
  4. Fine-tuning: After pre-training on all available data, the model is fine-tuned on a binary classification task to predict whether a student will show significant reading improvement over the academic year.

Key Insights and Results

The study revealed several critical insights into both predictive modeling and early childhood education:

  • Embedding Space > Raw Imputation: Learning to align embeddings of partial data proved far more effective than trying to reconstruct the missing raw values (as seen in models like VIME or TabNet).
  • Superior Accuracy: MaskMLP consistently outperformed standard baselines like XGBoost, LightGBM, and specialized tabular models. On the "Student Split" (testing on students from known schools), MaskMLP achieved an accuracy of 0.6919 and an AUC of 0.6899 for word identification tasks. The improvements over standard MLPs were statistically significant ($p = 0.0090$).
  • Interventions Clarify Data: Models performed roughly 10% better on the "Intervention Group" (students receiving extra support), with accuracy jumping to 0.7766. This suggests that structured educational interventions create more distinct, predictable patterns in student performance data.
Feature importance analysis highlighting Word Identification as the strongest predictor
  • The Power of Word Identification: Feature importance analysis (shown above) revealed that a student's initial "word identification" score at the start of the year is the single strongest predictor of their year-end success, closely followed by "word attack" skills (deciphering unfamiliar words).
  • Clustering Student Profiles: t-SNE visualizations demonstrated that MaskMLP pre-training naturally groups students into well-defined "profiles" (e.g., "Good initial word ID but low word attack") without needing explicit labels, helping the model generalize better.

Limitations and Future Work

While MaskMLP sets a new standard, challenges remain. The model still shows lower performance for "at-risk" students compared to the general population, likely due to the high heterogeneity of this group (which includes students facing diverse challenges like language barriers or learning disabilities). Additionally, there is a consistent trend of higher predictive performance in schools with higher socio-economic status, highlighting a need for more context-aware models to ensure equity.

Moving forward, the research team plans to transition toward "in-situ" deployment, providing teachers with real-time uncertainty estimates to help them decide when to trust the model's risk identification in the classroom.

Data and Code: The authors have made their benchmark and code available at https://ecri-data.github.io/.

Summary updated

BibTeX
@inproceedings{shangguan2024scalable,
  author = {Zhongkai Shangguan and Zanming Huang and Eshed Ohn-Bar and Ola Ozernov-Palchik and Derek Kosty and Michael Stoolmiller and Hank Fien},
  title = {{Scalable Early Childhood Reading Performance Prediction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure