ConfusionBench: An Expert-Validated Benchmark for Confusion Recognition and Localization in Educational Videos

Lu Dong, Xiao Wang, Mark Frank, Srirangaraj Setlur, Venu Govindaraju, Ifeoma NwoguarXiv 2026 · arXiv

TL;DR

ConfusionBench introduces a high-quality, expert-validated benchmark designed to evaluate how well AI can recognize and localize student confusion in educational videos. By combining open-source and proprietary Vision-Language Models (VLMs) with rigorous human expert validation, the authors address the noisy labels and coarse annotations plaguing previous datasets. Their baseline evaluations reveal a stark performance dichotomy between commercial and open-source VLMs when detecting subtle cognitive states.

What Problem Does This Paper Address?

Identifying when a student is confused is a holy grail for intelligent tutoring systems and automated educational interfaces. However, building AI models to do this requires reliable training and evaluation data—something existing datasets lack.

The authors point out three critical flaws in current benchmarks like DAiSEE:

  1. Noisy Labels: Crowdsourced annotations often mislabel clips; some marked "non-confusion" show clear visible distress, while others marked "confused" show no indicators at all.
  2. Coarse Temporal Annotations: Labels are typically applied to long 10-second blocks, even though actual confusion micro-expressions might only last 1–2 seconds.
  3. Lack of Expert Oversight: Large-scale crowdsourcing introduces massive subjective inconsistency without expert validation.
Noisy labels in existing datasets showing visible confusion cues despite negative labels

To bridge this gap, the paper introduces a systematic way to filter raw video data down to precise, high-confidence "golden clips" validated by cognitive and facial expression experts.

Methods

The authors developed a VLM-assisted multi-stage filtering pipeline to maximize data quality while minimizing the time required from expensive human experts:

  1. Fine-Grained Segmentation: Original 10-second video clips are chopped into non-overlapping 2-second segments to better isolate brief expressions.
  2. Model-Assisted Filtering: A lightweight open-source VLM (Qwen3-VL-4B-Instruct) performs a first-pass coarse screening. Next, a more powerful proprietary VLM (Google Gemini 3 Flash Preview) refines the selection using a confidence-aware majority voting strategy over multiple inference runs.
  3. Researcher Curation: Two researchers independently review the remaining clips using a strict behavioral protocol based on psychological literature (focusing on specific Facial Action Coding System cues like brow lowering and eyelid tightening).
  4. Expert Validation: Experts in facial expression analysis provide the final verdict (Yes, No, or Unsure).
The multi-stage filtering pipeline used to curate ConfusionBench

Finally, to enable long-video evaluation, the expert-validated golden clips are mapped back to their original long-form recordings, allowing the authors to propagate precise timestamp-level boundaries for confusion events.

Datasets

The paper introduces ConfusionBench, which is built on top of raw data from the widely used DAiSEE dataset but features completely overhauled annotations. It consists of two sub-datasets:

  • Balanced Confusion Recognition Dataset: Contains 450 2-second clips (224 expert-validated "Yes" confusion samples and 226 strictly verified "No" samples).
  • Long-Video Confusion Localization Dataset: Comprises ten 5-minute videos complete with exact start and end timestamps for every confusion event.
Comparison of coarse 10s clip duration vs fine-grained confusion time spans

Key Insights and Ideas

  • The Power of Hybrid Pipelines: Directly handing massive raw datasets to human experts is economically and logistically impossible. Using VLMs as an initial "sieve" effectively filters out the noise, letting experts focus their valuable time on high-probability, nuanced candidates.
  • VLM Behavioral Dichotomy: Commercial and open-source models exhibit opposite biases. Proprietary models (Gemini) are highly sensitive but prone to over-predicting confusion during transitional or ambiguous moments (high recall, lower precision). Conversely, open-source models (Qwen) are overly conservative, frequently missing subtle cues entirely (higher precision, very low recall).
  • Temporal Dilution Matters: Labeling a video at a coarse 10-second resolution dilutes the visual signal. Shifting the analysis window to 2 seconds is vital for capturing authentic, fleeting cognitive states.

Key Results

The authors conducted zero-shot baseline evaluations on ConfusionBench to see how well modern foundation models handle these tasks out of the box.

Clip-Level Recognition

  • Google Gemini 3 Flash Preview significantly outperformed the open-source alternative, achieving an F1-score of 0.8139 (Accuracy: 0.7978).
  • Qwen3-VL-4B-Instruct lagged behind with an F1-score of 0.6293 (Accuracy: 0.6911), severely hampered by a low recall of 0.5268.

Long-Video Localization

  • Gemini achieved a Micro-Average F1-score of 0.6934 and a temporal Intersection-over-Union (tIoU) of 0.5307. While it successfully identified that confusion was occurring, precisely pinning down the exact start and end boundaries proved challenging.
  • Qwen struggled heavily in the long-video format, yielding a Micro-Average F1-score of just 0.4530 and a low tIoU of 0.2928, confirming its weakness in temporal grounding.

Limitations and Future Work

While ConfusionBench provides a much-needed clean testbed, the authors note that its scale remains limited (450 clips and 10 long videos). Additionally, the current framework treats confusion as a simple binary state (present or absent). In reality, confusion is a dynamic process that scales in intensity (e.g., mild hesitation vs. deep frustration).

Future work will focus on expanding the dataset's scale, evaluating a broader range of multimodal models, and developing specialized localization frameworks capable of generating automated, actionable confusion reports for educators.

Summary updated

Abstract

Recognizing and localizing student confusion from video is an important yet challenging problem in educational AI. Existing confusion datasets suffer from noisy labels, coarse temporal annotations, and limited expert validation, which hinder reliable fine-grained recognition and temporally grounded analysis. To address these limitations, we propose a practical multi-stage filtering pipeline that integrates two stages of model-assisted screening, researcher curation, and expert validation to build a higher-quality benchmark for confusion understanding. Based on this pipeline, we introduce ConfusionBench, a new benchmark for educational videos consisting of a balanced confusion recognition dataset and a video localization dataset. We further provide zero-shot baseline evaluations of a representative open-source model and a proprietary model on clip-level confusion recognition, long-video confusion localization tasks. Experimental results show that the proprietary model performs better overall but tends to over-predict transitional segments, while the open-source model is more conservative and more prone to missed detections. In addition, the proposed student confusion report visualization can support educational experts in making intervention decisions and adapting learning plans accordingly. All datasets and related materials will be made publicly available on our project page.
BibTeX
@misc{dong2026confusionbench,
  author = {Lu Dong and Xiao Wang and Mark Frank and Srirangaraj Setlur and Venu Govindaraju and Ifeoma Nwogu},
  title = {{ConfusionBench: An Expert-Validated Benchmark for Confusion Recognition and Localization in Educational Videos}},
  year = {2026},
  eprint = {2603.17267},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.17267},
}
Figure