Towards Student Actions in Classroom Scenes: New Dataset and Baseline

Zhuolin Tan, Chenqiang Gao, Anyong Qin, Ruixin Chen, Tiecheng Song, Feng Yang, Deyu Meng

TL;DR

This paper introduces the Student Action Video (SAV) dataset, a high-resolution, multi-label benchmark containing over 4,300 video clips designed to capture complex, concurrent student behaviors in real-world classrooms. To tackle the unique challenges of dense crowds and tiny, occluded objects, the authors propose an enhanced Vision Transformer (ViT) baseline that integrates local feature aggregation and localized spatio-temporal attention, achieving state-of-the-art performance on the new benchmark.

What Problem Does This Paper Address?

While computer vision has made massive strides in action recognition and detection, existing benchmarks (like AVA or Kinetics) primarily focus on daily activities or sports. These datasets feature large, distinct actors against relatively simple backgrounds.

Classrooms present a completely different, highly chaotic environment. Students are packed densely together, often heavily occluded by desks or classmates in front of them. Their movements are subtle—such as the fine-grained difference between "touching a pen" and "taking notes"—and individuals frequently perform multiple actions simultaneously (e.g., "sitting," "looking forward," and "talking"). Existing educational datasets are often small, single-label, or too coarse-grained to be useful for real-world behavioral analysis. This paper fills this gap by providing both a challenging dataset and a specialized model architecture to handle it.

Datasets

The authors introduce the Student Action Video (SAV) dataset, which consists of 4,324 carefully trimmed 3-second video clips collected from surveillance footage across 758 distinct Chinese classrooms (ranging from kindergarten to middle school).

Key statistics and features of SAV include:

  • 15 Action Categories: Grouped into postural actions, sight actions, body motion, person-object interactions, and person-person interactions.
  • Multi-Person & Multi-Label: Multiple students are annotated per frame, with each student assigned between one and five concurrent action labels.
  • High Quality: Primarily 720p and 1080p resolutions, preserving fine details.
  • Crowded and Small-Scale: Bounding boxes in SAV typically occupy less than 1% of the total image frame, making detection exceptionally difficult compared to standard datasets.

The authors evaluate their method on both their new SAV dataset and the standard AVA dataset to demonstrate generalizability.

The stark contrast in object density and scale between SAV and traditional action detection datasets can be seen below:

Comparison of action detection datasets, highlighting the high density and small scale of subjects in SAV.

Methods

Standard Vision Transformers (ViTs) excel at capturing global context but often overlook minute local details, making them sub-optimal for dense, small-object environments like classrooms. To fix this, the authors propose an improved ViT baseline incorporating two novel modules:

  1. Local Relation Aggregator (LRA): Fused at the shallow layers of the network, the LRA uses depthwise convolutions and specialized relation aggregation layers to learn token affinities within small neighborhoods. This helps the model map out fine-grained interactions at a small scale.
  2. Window Enhanced Attention (WEA): Positioned at deeper layers, the WEA module uses a combination of average and max pooling to identify the local window regions with the strongest feature responses. It then applies localized spatio-temporal self-attention strictly to these regions, forcing the model to focus on discriminative tiny objects (like hands or eyes).

The overall pipeline extracts video features using this enhanced ViT, generates region proposals using a Faster R-CNN (ResNet-50) backbone on keyframes, and applies 3D RoI pooling to make final multi-label action classifications.

Key Insights and Ideas

  • Temporal Saturation: When optimizing the number of input frames, the authors found that performance peaks at 16 frames and then degrades. Because classroom actions change slowly or remain constant over short intervals, adding too many frames introduces informational redundancy that harms training.
  • Spatial Cues Trump Temporal Cues for Tiny Objects: In ablation studies of the WEA module, spatial-only attention outperformed temporal-only attention. This indicates that for fine-grained, small-scale action detection, capturing object morphology and scene structure is more critical than tracking temporal variations alone.
  • Data Frequency $\neq$ Model Ease: While a high volume of training data generally improves performance, action complexity matters more. Simple, distinct actions like "sitting" or rare but visually unique actions like "applauding" scored high, whereas highly frequent but visually similar actions like "touching something" vs. "taking notes" remained difficult for the model to differentiate.

Key Results

The proposed method achieves competitive results against several state-of-the-art architectures:

  • SAV Dataset: The model achieves a 67.90% mean Average Precision (mAP), outperforming the vanilla VideoMAE baseline (66.56%), MViTv2 (45.11%), and SlowFast R50 (34.97%).
  • AVA Dataset: It achieves 27.4% mAP, showing a slight edge over VideoMAE (26.7%). The smaller improvement on AVA is expected, as AVA contains larger-scale human actions where global context dominates over local details.
  • Ablation Impact: Adding the LRA module alone boosted the baseline ViT by 1.23% mAP, while the WEA module alone provided a 0.68% mAP increase, validating the complementary nature of both components.

Limitations and Future Work

The primary drawback of the proposed architecture is its computational complexity. Capturing local dynamic information across multiple dense objects via deep LRA and WEA modules significantly increases both the parameter count (172M vs. VideoMAE's 87M) and FLOPs (412G vs. VideoMAE's 180G).

The authors note that future work will focus on optimizing and simplifying the convolution operations within these modules to improve inference efficiency for practical, real-time deployment in educational settings.

Summary updated

Abstract

Analyzing student actions is an important and challenging task in educational research. Existing efforts have been hampered by the lack of accessible datasets to capture the nuanced action dynamics in classrooms. In this paper, we present a new multi-label Student Action Video (SAV) dataset, specifically designed for action detection in classroom settings. The SAV dataset consists of 4,324 carefully trimmed video clips from 758 different classrooms, annotated with 15 distinct student actions. Compared to existing action detection datasets, the SAV dataset stands out by providing a wide range of real classroom scenarios, high-quality video data, and unique challenges, including subtle movement differences, dense object engagement, significant scale differences, varied shooting angles, and visual occlusion. These complexities introduce new opportunities and challenges to advance action detection methods. To benchmark this, we propose a novel baseline method based on a visual transformer, designed to enhance attention to key local details within small and dense object regions. Our method demonstrates excellent performance with a mean Average Precision (mAP) of 67.9% and 27.4% on the SAV and AVA datasets, respectively. This paper not only provides the dataset but also calls for further research into AI-driven educational tools that may transform teaching methodologies and learning outcomes. The code and dataset are released at https://github.com/Ritatanz/SAV.
BibTeX
@article{tan2025towards,
  author = {Zhuolin Tan and Chenqiang Gao and Anyong Qin and Ruixin Chen and Tiecheng Song and Feng Yang and Deyu Meng},
  title = {{Towards Student Actions in Classroom Scenes: New Dataset and Baseline}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3590899},
}

Loading related content...

Figure