Relating Children's Automatically Detected Facial Expressions to Their Behavior in RoboTutor

Mayank Saxena, Rohith Krishnan Pillai, Jack MostowAAAI 2018 · DOI

Introduction

Intelligent Tutoring Systems (ITS) are designed to maintain student engagement, yet many current implementations are reactive in a delayed manner, adjusting lessons only after analyzing session data. Furthermore, much of the existing research on affective computing in education focuses on adults in controlled laboratory settings. This research addresses these gaps by exploring whether a child's imminent behavior can be predicted in real-time based on their facial expressions, specifically within the context of RoboTutor, an open-source Android tablet tutor for literacy and numeracy.

The study focuses on children aged 6-12, many of whom have limited prior schooling or technology access. The goal is to move beyond general engagement metrics to predict specific actions, such as prematurely exiting an activity, allowing for more timely and effective pedagogical interventions.

Methodology

The researchers utilized data from RoboTutor sessions to map facial expressions to pedagogical behaviors:

  1. Affective State Detection: The OpenFace toolkit extracted Facial Action Units (FAUs) from the tablet's front-facing camera.
  2. Emotion Mapping: These FAUs were mapped to six pedagogically relevant emotions: boredom, confusion, frustration, surprise, delight, and neutral.
  3. Data Integration: Detected emotions were synchronized with application log files recording user actions (e.g., button presses, activity completion, and answer correctness).
  4. Analysis: Statistical correlation and visualization identified links between these emotional states and subsequent student behaviors.

The dataset comprised 17 unique session pairs ($N=17$), with sessions lasting 20 to 30 minutes. Activities included racing games, writing exercises, and read-along stories.

Key Insights

Delight as a Predictor of Completion

A significant finding was the relationship between expressed delight and task persistence. Students who successfully completed activities exhibited significantly higher median levels of delight compared to those who did not ($p < 0.05$). This suggests that positive affective reinforcement is a strong indicator of a student's likelihood to finish a task.

Figure 1: Comparison of delight levels between successful (1) and unsuccessful (0) activity completions.

The "Flow" of Neutrality in Performance

In the "bubble pop" activity, researchers observed a counterintuitive trend regarding performance. While one might expect positive emotions to correlate with success, the only state that positively correlated with correctness was "neutral." Every other emotion—including delight—showed a negative correlation with performance. This indicates that deep cognitive processing and "flow" often manifest as a neutral facial expression, whereas high-arousal emotions (even positive ones) may interfere with the focus required for certain tasks.

Figure 2: Correlation heatmap between emotional states and correctness in the "bubble pop" activity, showing neutral as the sole positive correlate.

Surprise as a "Quit" Signal

While boredom was a frequent state, it was not the primary predictor of quitting. Instead, "surprise" emerged as a statistically significant indicator that a student was about to hit the "back" button. This suggests that a sudden mismatch in expectations—perhaps due to a difficult task or confusing interface—triggers a child to exit an activity prematurely.

Conclusion and Future Work

The study demonstrates that specific affective states are statistically significant markers for imminent student behavior. High delight predicts completion, while surprise often precedes quitting. Notably, a neutral expression is the best correlate for correctness in focused tasks.

The primary limitation of this work is the small sample size ($N=17$). Future work involves integrating these real-time predictive capabilities into RoboTutor’s intervention policy. By adjusting the tutor's strategy the moment a "quit" behavior is predicted, the system could significantly reduce session dropout rates and improve learning outcomes.

Summary updated

BibTeX
@inproceedings{saxena2018relating,
  author = {Mayank Saxena and Rohith Krishnan Pillai and Jack Mostow},
  title = {{Relating Children's Automatically Detected Facial Expressions to Their Behavior in RoboTutor}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12190},
}
Figure