Identifying At-Risk K-12 Students in Multimodal Online Environments: A Machine Learning Approach

Hang Li, Wenbiao Ding, Songfan Yang, Zitao Liu

Abstract

Predicting student dropout in K-12 online tutoring presents unique challenges compared to traditional Massive Open Online Courses (MOOCs), including lower dropout rates, high interactivity across multiple modalities, and significant time sensitivity. This paper proposes a machine learning framework utilizing a Gradient Boosting Decision Tree (GBDT) model to identify at-risk students. By integrating multimodal in-class features with out-of-class activity logs and implementing a novel time-aware data augmentation strategy, the system achieves a 70% detection rate in a real-world production environment.

Introduction

While dropout prediction is well-studied in MOOCs, the K-12 online tutoring landscape is distinct. MOOCs typically involve adult learners with high attrition rates (70-93%). In contrast, K-12 tutoring is often a paid service with dropout rates below 20%. The authors identify four primary challenges in this domain: multimodal interactions (video, voice, text), high variability in student learning history length, time sensitivity (the "recency effect"), and severe data imbalance.

Methodology

The proposed framework centers on a GBDT model, leveraging a comprehensive feature set and a specialized data augmentation technique.

Multimodal Feature Engineering

The model incorporates three distinct categories of features to capture a holistic view of student engagement:

  1. In-class Features: Derived from classroom recordings, including prosodic (loudness, pitch), linguistic (speech speed, sentence length), and interaction metrics (Q&A frequency).
  2. Out-of-class Features: Service-side data such as pre-class negotiations, discount ratios, and post-class rescheduling frequency.
  3. Time-variant Features: Statistics calculated over specific lookback windows (e.g., activity in the last 7 days) to capture temporal trends.
Table 1: Categorization of features used in the prediction model, including in-class, out-of-class, and time-variant types.

Time-Aware Data Augmentation

To address data imbalance, the researchers introduced "pseudo-positive" samples. This approach assumes that students who eventually drop out exhibit "at-risk" behaviors in the preceding days. These samples are weighted using a convex function, ensuring the model prioritizes behaviors immediately prior to dropout, effectively modeling the "recency effect."

Results and Discussion

The study utilized a dataset of 3,922 students from a K-12 online tutoring platform, covering over 338,000 observation pairs with a 16.16% dropout rate.

Model Performance

Offline evaluations demonstrated that the proposed GBDT model consistently outperformed baseline methods like Logistic Regression (LR), Random Forest (RF), and Decision Trees (DT). Notably, the model maintained superior performance across various prediction horizons, ranging from 1 to 14 days ahead.

Figure 1: Comparison of AUC scores across different machine learning models over varying multi-step ahead prediction time spans.

Key Insights

  • The Recency Effect: Emphasizing recent behavior through convex weighting significantly improved predictive accuracy compared to linear or concave models.
  • Feature Importance: Surprisingly, out-of-class service features (e.g., rescheduling frequency) were more predictive than in-class multimodal features, suggesting that administrative friction is a major driver of attrition in paid K-12 contexts.
  • Prediction Horizon: Model performance is highest for short-term predictions (AUC 0.8262 for 1-day ahead) and gradually declines as the window extends to 14 days (AUC 0.7430), reflecting the volatility of student intent.

Conclusion

The framework's deployment in a live production environment resulted in the successful identification of over 70% of students who eventually dropped out, enabling timely interventions. Future work will explore Deep Neural Networks to capture more complex non-linear relationships and aim to improve ASR accuracy to refine linguistic feature extraction.

Summary updated

BibTeX
@inproceedings{li2020identifying,
  author = {Hang Li and Wenbiao Ding and Songfan Yang and Zitao Liu},
  title = {{Identifying At-Risk K-12 Students in Multimodal Online Environments: A Machine Learning Approach}},
  booktitle = {EDM},
  year = {2020},
}
Figure