TL;DR
This paper introduces ORBITS, a novel video-based learning platform that leverages knowledge tracing to personalize the student experience. By analyzing student interactions with video content, ORBITS aims to predict learning needs and adapt content delivery, showing promising initial results in improving student engagement and satisfaction.
What Problem Does This Paper Address?
The COVID-19 pandemic accelerated the adoption of online learning, but challenges like student disengagement and difficulty in personalizing learning experiences persist. While video-based learning (VBL) has shown potential to improve engagement and learning outcomes, there's a gap in leveraging advanced AI techniques like knowledge tracing to create truly personalized VBL platforms, especially at the postgraduate level. Existing approaches often struggle with scalability and accurately predicting student mastery over time, particularly with complex, long sequences of learning activities.
Methods
The ORBITS platform is built upon a novel knowledge tracing model that goes beyond standard deep learning approaches. Key methodological contributions include:
- Question to Topic Mapping: Instead of processing data at the question level, which leads to high dimensionality, ORBITS maps questions to topics. This reduces dimensionality and allows for topic-level knowledge state tracking.
- Representative Subset of Feature Space: To handle the large feature space of topics, ORBITS employs a sampling method to select a representative subset of features, improving model performance and scalability.
- Topic-Based Encoding: This technique creates a matrix of topic interactions, capturing dependencies between topics and improving prediction accuracy by understanding how topics relate to each other.
- Student Knowledge Context: The model incorporates the chronological order of student interactions and emphasizes recent knowledge states to better capture the learning context, addressing the "forgetting" aspect of cognitive load.
- LSTM-based Architecture: The core of the model utilizes Long Short-Term Memory (LSTM) networks, architected to process sequences both horizontally across topics and vertically across learning time per topic, enabling a unique approach to knowledge tracing.
- ML Ops Pipeline: An end-to-end machine learning operations pipeline was developed for efficient large-scale training, hyperparameter tuning, and deployment of models.
Datasets
The paper mentions using several publicly available datasets for evaluating their knowledge tracing techniques:
- ASSIST2009
- ASSIST2015
- ASSISTments Challenge
- STATISTICS
These datasets are standard benchmarks in the knowledge tracing field and contain student interaction data with educational content. The study also involved an exploratory case study with postgraduate students at Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI).
Key Insights and Ideas
- Topic-level tracking is more scalable: Moving from question-level to topic-level analysis is crucial for handling the vastness of educational content and scaling knowledge tracing models.
- Context matters: Incorporating the temporal and sequential context of student learning is vital for accurate predictions, especially considering how students forget and build upon prior knowledge.
- Personalization leads to satisfaction: Initial qualitative feedback strongly suggests that personalized learning experiences, facilitated by AI, significantly enhance student satisfaction and engagement.
- Beyond accuracy: Stability is key for ITS: While prediction accuracy is important, the stability of model parameters is critical for providing actionable insights and enabling adaptive behavior in intelligent tutoring systems.
Key Results
- Qualitative Feedback: Student feedback on the ORBITS platform was overwhelmingly positive, highlighting its ease of use, responsiveness, and usefulness. Students appreciated features like transcript searching, speed-up capabilities, and customized reports on their understanding.
- Quantitative Benchmarking: Table 1 shows that their proposed model, incorporating topic-based encoding and student context, achieved competitive or improved AUC scores compared to baseline BKT and DKT models on standard datasets. For instance, on ASSIST2009, their "Self-Attention" model achieved an AUC of 84.20 ± 0.10, outperforming BKT (0.630) and DKT (81.81 ± 0.10).
- Engagement and Perception: The exploratory case study indicated that students using ORBITS were more engaged and satisfied with their learning experience compared to those who did not.
Limitations and Future Work
- Limited Sample Size: The study was based on a small cohort of postgraduate students from a single master's course. Generalizability of the findings is therefore limited.
- AI University Bias: The positive opinion of AI-powered technologies might be influenced by the fact that the study was conducted at an AI university.
- Course Content Influence: The impact of course content on student experience and the system's effectiveness remains unclear.
Future work suggested includes:
- Larger-scale studies: Testing the system with larger student samples and across different disciplines to improve generalizability.
- Quantitative data collection: Collecting final student grades to compare outcomes between students who used ORBITS and those who did not.
- Investigating influencing factors: Further research into how factors like Perceived Ease of Use (PEoU) and Perceived Usefulness (PU) predict Attitude towards Usage (AtU) of ORBITS.
- Testing across multiple courses and lecturers: To understand the influence of course content and teaching styles.
- Testing at non-AI universities: To mitigate potential bias from the study's current location.
Summary updated