Abstract
Analyzing student actions is an important and challenging task in educational research. Existing efforts have been hampered by the lack of accessible datasets to capture the nuanced action dynamics in classrooms. In this paper, we present a new multi-label student action video (SAV) dataset for complex classroom scenes. The dataset consists of 4,324 carefully trimmed video clips from 758 different classrooms, each labeled with 15 different actions displayed by students in classrooms. Compared to existing behavioral datasets, our dataset stands out by providing a wide range of real classroom scenarios, high-quality video data, and unique challenges, including subtle movement differences, dense object engagement, significant scale differences, varied shooting angles, and visual occlusion. The increased complexity of the dataset brings new opportunities and challenges for benchmarking action detection. Innovatively, we also propose a new baseline method, a visual transformer for enhancing attention to key local details in small and dense object regions. Our method achieves excellent performance with mean Average Precision (mAP) of 67.9% and 27.4% on SAV and AVA, respectively.
Introduction
While action recognition has advanced significantly with datasets like Kinetics and AVA, these benchmarks primarily focus on sports or daily life. Educational research lacks high-quality, large-scale datasets specifically for the classroom. Classroom scenes are notoriously difficult for computer vision because they feature dense object engagement, significant scale differences between front and back rows, subtle movements (e.g., distinguishing "touching a pen" from "taking notes"), and frequent visual occlusion by desks or peers.
The Student Action Video (SAV) Dataset
The core contribution of this work is the Student Action Video (SAV) dataset. It features 4,324 trimmed 3-second clips from 758 different classrooms. It includes 15 fine-grained action labels categorized into five types: postural, sight, person-object, body-motion, and person-person interactions.
Through extensive observation, the authors determined that 3-second clips represent a "sweet spot"—long enough to capture postural changes like sitting or standing, but short enough to isolate brief interactions like raising a hand without introducing too much redundant, static data.
Methodology
The authors propose an improved Vision Transformer (ViT) architecture designed to "zoom in" on local details that standard transformers often overlook. The model introduces two primary modules:
- Local Relation Aggregator (LRA): Standard ViTs excel at global context but struggle with local dynamics. The LRA uses depthwise convolutions and a relation aggregator to learn token affinities at a shallow level, helping the model understand small-scale interactions.
- Window Enhanced Attention (WEA): This module identifies the most informative window regions within a video token. By applying spatio-temporal attention specifically to these high-energy regions, the model can better detect discriminative tiny objects (like a hand raising or a pen moving).
The pipeline integrates these into a backbone that feeds into a Region Proposal Network (RPN) and RoI Pooling for final action classification.
Experimental Results
The proposed method outperformed several established models in the field:
- SAV Dataset: Achieved 67.90% mAP, significantly higher than VideoMAE (66.56%) and MViT_B (41.45%).
- AVA Dataset: Achieved 27.4% mAP, demonstrating competitive performance even on non-classroom data.
Ablation studies confirmed the effectiveness of the new modules: the LRA module alone improved mAP by 1.23%, while the WEA module added another 0.68%. This supports the strategy of focusing on local details for dense classroom scenes. Notably, performance peaked at 16 sampled frames; adding more frames actually decreased accuracy as excessive frames introduced noise from static periods.
Conclusion
This paper introduces a challenging new benchmark for student action recognition and a baseline model that addresses the specific difficulties of classroom environments. While occlusion remains a significant hurdle, the SAV dataset and the proposed ViT enhancements provide a strong foundation for future AI-driven educational tools. Future work will focus on improving inference efficiency and handling high intra-class diversity in complex human activities.
Summary updated