Generalisable Methods for Early Prediction in Interactive Simulations for Education

Jade Maï Cock, Mirko Marras, Christian Giang, Tanja Käser

Introduction

Interactive simulations, such as those developed by the PhET Interactive Simulations project, have become foundational tools in modern science education. By allowing students to manipulate variables like voltage, concentration, or light absorbance in a virtual environment, these tools facilitate inquiry-based learning. However, the open-ended nature of these simulations presents a challenge: without real-time guidance, students may fail to grasp the underlying scientific principles or become stuck in unproductive exploration patterns.

To address this, researchers have focused on "early prediction"—the ability of an AI system to determine a student's conceptual understanding while they are still interacting with the simulation. This paper introduces a generalisable deep learning framework designed to provide accurate, early-stage predictions across different scientific domains and student populations.

The Challenge: Moving Beyond Post-Hoc Analysis

Traditionally, educational data mining has relied on a-posteriori analysis, where student performance is evaluated after a session is completed. While useful for research, this approach is insufficient for real-time intervention. For an AI tutor to be effective, it must identify a struggling student within the first few minutes of interaction.

Furthermore, existing models are often "brittle," meaning they are tailored to a specific simulation or a specific demographic and fail to generalise when applied to new contexts. The goal of this research was to develop a method that is both early-acting and robust enough to handle diverse educational settings.

Methodology

The researchers developed a pipeline that transforms raw clickstream data into a structured format suitable for sequential deep learning. The methodology consists of three primary components:

  1. State-Action Encoding: Rather than simply tracking clicks, the model encodes the state of the simulation (e.g., which tools are active) alongside the action performed (e.g., adjusting a slider). This provides the model with the necessary context to interpret the student's intent.
  2. Sequential Modeling with GRUs: The framework utilizes Gated Recurrent Units (GRUs), a type of Recurrent Neural Network (RNN) optimized for time-series data. Unlike traditional classifiers, GRUs can maintain a "memory" of the sequence of a student's actions, which is critical for understanding the trajectory of their inquiry.
  3. Self-Attention Mechanisms (SA-GRU): By adding an attention layer to the GRU, the model can dynamically weigh the importance of different moments in the interaction. This not only improves predictive accuracy but also enhances the interpretability of the model’s decisions.

To validate the approach, the researchers compared the SA-GRU against a Random Forest (RF) baseline, a common machine learning model used in previous educational research.

Experimental Setup and Datasets

To test the generalisability of the framework, the study utilized two distinct datasets representing different subjects, demographics, and geographic locations:

  • Beer’s Law Lab: This dataset involved 254 vocational school apprentices in Europe interacting with a chemistry simulation focused on light absorbance.
  • Capacitor Lab: This dataset involved 193 undergraduate physics students at a US university exploring the mechanics of capacitors.

By testing on these disparate groups, the researchers could evaluate whether the model's feature engineering and architecture could transcend specific classroom contexts.

Key Insights

The analysis revealed several critical findings regarding how students learn in virtual environments:

  • Context is Key: The attention weights indicated that the state of the simulation was often a stronger predictor of understanding than the specific actions taken. Knowing where a student was looking or which tools were active provided more insight than the mere fact that they moved a slider.
  • The Value of "Thinking Breaks": One of the most intriguing findings was the role of inactivity. The model identified that students who took brief pauses during the early stages of exploration were often more successful. These "thinking breaks" likely represent moments of reflection and inquiry-based planning, whereas continuous, rapid clicking often signaled a lack of strategy.
  • The Importance of Sequence: Shallow models like Random Forest often aggregate data (e.g., total time spent), losing the temporal context. The GRU’s ability to process the order of operations allowed it to identify unproductive inquiry patterns much earlier than traditional methods.

Results

The deep learning framework demonstrated significant improvements over the baseline across both datasets:

  • Accuracy Gains: In the Beer’s Law dataset, the SA-GRU achieved an AUC of 0.77, representing a 10% improvement over the Random Forest baseline (0.70).
  • Early Prediction Performance: The model's most significant advantage appeared in the early stages of interaction. When analyzing only the first 30 interactions, the sequential models outperformed the baseline by 15%. This is the critical window where an automated intervention would be most effective.
  • Cross-Domain Robustness: The model performed exceptionally well on the Capacitor Lab dataset (AUC ~0.96), confirming that the state-action encoding approach is highly generalisable across different scientific topics and student levels.

Conclusion and Future Directions

This research demonstrates that sequential deep learning, combined with state-aware feature engineering, can provide the early and accurate predictions necessary for real-time adaptive learning. However, several challenges remain. The current models use binary labels ("limited" vs. "advanced" understanding), which may oversimplify the learning process. Additionally, preliminary analysis suggested potential algorithmic biases regarding gender and geography that must be addressed before deployment.

Future work will focus on moving from prediction to intervention—designing AI-driven feedback that is timely, helpful, and supportive of the student's inquiry process without being overly intrusive.

Summary updated

Abstract

Interactive simulations allow students to discover the underlying principles of a scientific phenomenon through their own exploration. Unfortunately, students often struggle to learn effectively in these environments. Classifying students' interaction data in the simulations based on their expected performance has the potential to enable adaptive guidance and consequently improve students' learning. Previous research in this field has mainly focused on a-posteriori analyses or investigations limited to one specific predictive model and simulation. In this paper, we investigate the quality and generalisability of models for an early prediction of conceptual understanding based on clickstream data of students across interactive simulations. We first measure the students' conceptual understanding through their in-task performance. Then, we suggest a novel type of features that, starting from clickstream data, encodes both the state of the simulation and the action performed by the student. We finally propose to feed these features into GRU-based models, with and without attention, for prediction. Experiments on two different simulations and with two different populations show that our proposed models outperform shallow learning baselines and better generalise to different learning environments and populations. The inclusion of attention into the model increases interpretability in terms of effective inquiry. The source code is available on Github (https://github.com/epfl-ml4ed/beerslaw-lab.git).
BibTeX
@inproceedings{cock2022generalisable,
  author = {Jade Maï Cock and Mirko Marras and Christian Giang and Tanja Käser},
  title = {{Generalisable Methods for Early Prediction in Interactive Simulations for Education}},
  booktitle = {EDM},
  year = {2022},
}
Figure