Towards Student Actions in Classroom Scenes: New Dataset and Baseline

Zhuolin Tan, Chenqiang Gao, Anyong Qin, Ruixin Chen, Tiecheng Song, Feng Yang, Deyu Meng

Abstract

Analyzing student actions is an important and challenging task in educational research. Existing efforts have been hampered by the lack of accessible datasets to capture the nuanced action dynamics in classrooms. In this paper, we present a new multi-label student action video (SAV) dataset for complex classroom scenes. The dataset consists of 4,324 carefully trimmed video clips from 758 different classrooms, each labeled with 15 different actions displayed by students in classrooms. Compared to existing behavioral datasets, our dataset stands out by providing a wide range of real classroom scenarios, high-quality video data, and unique challenges, including subtle movement differences, dense object engagement, significant scale differences, varied shooting angles, and visual occlusion. The increased complexity of the dataset brings new opportunities and challenges for benchmarking action detection. Innovatively, we also propose a new baseline method, a visual transformer for enhancing attention to key local details in small and dense object regions. Our method achieves excellent performance with mean Average Precision (mAP) of 67.9% and 27.4% on SAV and AVA, respectively.

Introduction

While action recognition has advanced significantly with datasets like Kinetics and AVA, these benchmarks primarily focus on sports or daily life. Educational research lacks high-quality, large-scale datasets specifically for the classroom. Classroom scenes are notoriously difficult for computer vision because they feature dense object engagement, significant scale differences between front and back rows, subtle movements (e.g., distinguishing "touching a pen" from "taking notes"), and frequent visual occlusion by desks or peers.

Comparison of action sequences and dense classroom environments.

The Student Action Video (SAV) Dataset

The core contribution of this work is the Student Action Video (SAV) dataset. It features 4,324 trimmed 3-second clips from 758 different classrooms. It includes 15 fine-grained action labels categorized into five types: postural, sight, person-object, body-motion, and person-person interactions.

Examples of multi-label action annotations in the SAV dataset.

Through extensive observation, the authors determined that 3-second clips represent a "sweet spot"—long enough to capture postural changes like sitting or standing, but short enough to isolate brief interactions like raising a hand without introducing too much redundant, static data.

Methodology

The authors propose an improved Vision Transformer (ViT) architecture designed to "zoom in" on local details that standard transformers often overlook. The model introduces two primary modules:

  1. Local Relation Aggregator (LRA): Standard ViTs excel at global context but struggle with local dynamics. The LRA uses depthwise convolutions and a relation aggregator to learn token affinities at a shallow level, helping the model understand small-scale interactions.
  2. Window Enhanced Attention (WEA): This module identifies the most informative window regions within a video token. By applying spatio-temporal attention specifically to these high-energy regions, the model can better detect discriminative tiny objects (like a hand raising or a pen moving).

The pipeline integrates these into a backbone that feeds into a Region Proposal Network (RPN) and RoI Pooling for final action classification.

Experimental Results

The proposed method outperformed several established models in the field:

  • SAV Dataset: Achieved 67.90% mAP, significantly higher than VideoMAE (66.56%) and MViT_B (41.45%).
  • AVA Dataset: Achieved 27.4% mAP, demonstrating competitive performance even on non-classroom data.

Ablation studies confirmed the effectiveness of the new modules: the LRA module alone improved mAP by 1.23%, while the WEA module added another 0.68%. This supports the strategy of focusing on local details for dense classroom scenes. Notably, performance peaked at 16 sampled frames; adding more frames actually decreased accuracy as excessive frames introduced noise from static periods.

Conclusion

This paper introduces a challenging new benchmark for student action recognition and a baseline model that addresses the specific difficulties of classroom environments. While occlusion remains a significant hurdle, the SAV dataset and the proposed ViT enhancements provide a strong foundation for future AI-driven educational tools. Future work will focus on improving inference efficiency and handling high intra-class diversity in complex human activities.

Summary updated

Abstract

Analyzing student actions is an important and challenging task in educational research. Existing efforts have been hampered by the lack of accessible datasets to capture the nuanced action dynamics in classrooms. In this paper, we present a new multi-label student action video (SAV) dataset for complex classroom scenes. The dataset consists of 4,324 carefully trimmed video clips from 758 different classrooms, each labeled with 15 different actions displayed by students in classrooms. Compared to existing behavioral datasets, our dataset stands out by providing a wide range of real classroom scenarios, high-quality video data, and unique challenges, including subtle movement differences, dense object engagement, significant scale differences, varied shooting angles, and visual occlusion. The increased complexity of the dataset brings new opportunities and challenges for benchmarking action detection. Innovatively, we also propose a new baseline method, a visual transformer for enhancing attention to key local details in small and dense object regions. Our method achieves excellent performance with mean Average Precision (mAP) of 67.9\% and 27.4\% on SAV and AVA, respectively. This paper not only provides the dataset but also calls for further research into AI-driven educational tools that may transform teaching methodologies and learning outcomes. The code and dataset will be released at https://github.com/Ritatanz/SAV.
BibTeX
@article{tan2025towards,
  author = {Zhuolin Tan and Chenqiang Gao and Anyong Qin and Ruixin Chen and Tiecheng Song and Feng Yang and Deyu Meng},
  title = {{Towards Student Actions in Classroom Scenes: New Dataset and Baseline}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3590899},
}
Figure