TL;DR
This paper introduces a novel hierarchical multi-instance learning (MIL) framework to automatically assess student engagement in online learning videos using only video-level labels. By learning relationships between frames, clips, and the entire video, their method can predict engagement at arbitrary temporal granularities, addressing the costly challenge of fine-grained annotation.
What Problem Does This Paper Address?
Massive Open Online Courses (MOOCs) offer unprecedented access to education, but suffer from low student retention rates. To combat this, personalized, just-in-time interventions are crucial, which in turn require accurate assessment of student engagement. Manually annotating engagement levels for every frame or short clip in lengthy online course videos is prohibitively expensive and time-consuming. Existing methods often focus on assessing engagement for the entire video, missing the opportunity to identify engagement dips within specific segments. This paper tackles the challenge of learning fine-grained (clip-level) engagement assessment using only coarse, video-level labels, bridging the gap between expensive detailed annotations and the need for dynamic, granular engagement insights.
Methods
The core of the proposed solution is a Hierarchical Temporal Multi-Instance Learning (H-TMIL) framework. This framework consists of two main modules:
- Bottom Temporal Multi-Instance Learning (B-TMIL): This module operates at the clip level. It takes sequences of frame-level features (extracted from facial landmarks, head pose, and body pose) as input. Unlike standard MIL, it incorporates a temporal aspect by using a bidirectional LSTM followed by a self-attention mechanism. This allows it to adaptively weigh the importance of different frames within a clip to derive a robust clip-level engagement representation.
- Top Temporal Multi-Instance Learning (T-TMIL): This module aggregates the clip-level representations to assess the engagement of the entire video. It also employs a MIL approach, but treats clips as instances and the video as a bag. This module learns a weighted combination of clip-level features, allowing for a more flexible and robust video-level engagement representation than simple averaging.
The entire H-TMIL network is trained end-to-end using only video-level labels. A joint loss function is employed, comprising a global supervision loss (comparing predicted video-level engagement with the ground truth) and a local supervision loss (comparing the average of predicted clip-level engagements with the video-level label). This hierarchical structure and joint loss enable the model to implicitly learn clip-level engagement scores while being supervised by video-level labels.
Datasets
The authors introduce a new dataset called the Online Course Studies (OCS) Data set. This dataset comprises 236 videos of 59 subjects watching healthcare-topic MOOCs. Each video is approximately 30 minutes long. The dataset is annotated with engagement scores ranging from 0 to 1, derived from a combination of self-scoring, annotator scoring, and test scores, normalized to the [0, 1] range.
Key Insights and Ideas
- Leveraging Video-Level Labels for Fine-Grained Assessment: The most significant insight is the demonstration that it's possible to learn meaningful clip-level engagement predictions using only video-level labels. This bypasses the need for expensive, frame-by-frame or clip-by-clip annotations.
- Hierarchical Structure for Temporal Dependencies: The hierarchical MIL approach effectively captures temporal dependencies at different granularities: frame-to-clip (B-TMIL) and clip-to-video (T-TMIL). This is crucial for understanding how engagement evolves over time.
- Self-Attention for Adaptive Feature Aggregation: The use of self-attention within the B-TMIL module allows the model to dynamically focus on the most informative frames within a clip, leading to more accurate clip-level representations. This is shown to be superior to simple mean pooling or relying solely on the last hidden state of an LSTM.
- Joint Local and Global Supervision: Combining local (clip-level) and global (video-level) supervision in the loss function leads to better overall performance, suggesting that constraining both levels of prediction simultaneously is beneficial.
Key Results
- The proposed H-TMIL method achieves state-of-the-art performance on the OCS dataset, outperforming several peer methods in terms of Mean Absolute Error (MAE) and Mean Squared Error (MSE) for both video-level and clip-level engagement assessment.
- For video-level assessment, the H-TMIL method achieved an MAE of 0.094 and MSE of 0.0158.
- For clip-level assessment, the H-TMIL method achieved an MAE of 0.098 and MSE of 0.017.
- Visualizations show that the predicted clip-level engagement trends closely follow the ground truth labels, even for videos with varying engagement levels.
- Ablation studies confirm the effectiveness of the TMIL modules and the joint local/global supervision strategy.
Limitations and Future Work
- Limitations: The paper notes that while their model performs well, the OCS dataset has a relatively large float in the range of labels for one video, which might affect performance on that specific instance. The feature extraction models used can sometimes produce missing outputs, requiring interpolation.
- Future Work: The authors suggest two main directions:
- More Effective Structure: Unifying feature extraction and assessment into a single end-to-end model and exploring more advanced structures to capture high-level semantic information more effectively.
- Practicality: Building a real-time system for online engagement assessment and demonstrating the model's versatility in other scenarios, such as evaluating user interest in advertisements or detecting fatigue while driving.
Summary updated