Who Said What (WSW 2.0)? Enhanced Automated Analysis of Preschool Classroom Speech

Anchen Sun, Tiantian Feng, Gabriela Gutierrez, Juan J. Londono, Anfeng Xu, Batya Elbaum, Shrikanth Narayanan, Lynn K. Perry, Daniel S. Messinger

Introduction

Understanding teacher-child interactions in preschool settings is fundamental to early childhood education research. However, the sheer volume of audio data generated in these environments creates a massive technical bottleneck. Traditional manual transcription is labor-intensive, often requiring hours of human effort for a single minute of audio. While commercial solutions exist, they often lack transparency and struggle with the high-noise, multi-speaker dynamics of a classroom.

WSW 2.0 is an open-source automated framework designed to overcome these challenges. By integrating state-of-the-art deep learning models for speaker classification and speech transcription, it enables large-scale, longitudinal analysis of classroom discourse with high reliability.

Methodology

The WSW 2.0 framework utilizes a multi-stage automated pipeline to process raw audio into actionable linguistic data.

The WSW 2.0 automated pipeline and reliability evaluation workflow.

The system consists of two primary deep learning components:

  1. Speaker Classification: An Egocentric Speaker Classification (ESC) model based on wav2vec2 determines whether an utterance was produced by a child or a teacher.
  2. Speech Transcription: OpenAI’s Whisper model (specifically the large-v2 variant) is employed for automated speech recognition (ASR).
  3. Feature Extraction: The pipeline automatically calculates developmental indices such as Mean Length of Utterance (MLU), Lexical Diversity (LD), and conversational dynamics like question frequency and "responsivity" (the proportion of child utterances followed by a teacher response within 2.5 seconds).

Datasets and Validation

The framework was validated using a Reliability Dataset of 235 minutes of expert-coded audio. To demonstrate scalability, the researchers applied the tool to a Longitudinal Scalability Dataset comprising 1,592 hours of classroom recordings collected over two academic years. This massive dataset, containing over 1.4 million utterances, was processed in approximately two weeks—a task that would be impossible for human coders.

Key Insights

A notable finding was the performance difference between Whisper model versions. Contrary to expectations, Whisper large-v2 outperformed large-v3 for child-specific metrics, particularly child MLU. This highlights the importance of selecting models based on specific acoustic tasks rather than just version numbers.

Furthermore, the study addressed the "Child Gap" in ASR. While the system is highly effective, the Word Error Rate (WER) for teachers (.119) remains significantly lower than for children (.238), reflecting the inherent difficulty of processing child speech in noisy environments.

Results and Reliability

The WSW 2.0 framework demonstrated high reliability across various linguistic and conversational metrics when compared to expert human annotations.

Intraclass Correlation (ICC) results comparing WSW 2.0 and WSW 1.0 against expert human coding.

Key performance metrics include:

  • Classification Accuracy: An overall accuracy of 84.6% (weighted F1 of .845) in identifying speakers.
  • Transcription Reliability: Intraclass Correlation (ICC) values for features like question-asking and MLU ranged from .64 to .98, indicating strong agreement with human experts.
  • Enhanced Performance: As shown in the comparison with the previous version (WSW 1.0), the new ESC model significantly improves the reliability of child-specific metrics, such as question utterances and response proportions.

Limitations and Future Directions

Despite its strengths, WSW 2.0 faces challenges with overlapping speech (cross-talk) and the persistent performance gap in child ASR. Future work will focus on improving semantic analysis—measuring the topical alignment between teachers and children—and further refining models for the unique acoustic profiles of young learners.

As an open-source tool, WSW 2.0 provides a transparent and scalable foundation for researchers to design and evaluate educational interventions that support early language development at scale.

Summary updated

Abstract

This paper introduces an automated framework WSW2.0 for analyzing vocal interactions in preschool classrooms, enhancing both accuracy and scalability through the integration of wav2vec2-based speaker classification and Whisper (large-v2 and large-v3) speech transcription. A total of 235 minutes of audio recordings (160 minutes from 12 children and 75 minutes from 5 teachers), were used to compare system outputs to expert human annotations. WSW2.0 achieves a weighted F1 score of .845, accuracy of .846, and an error-corrected kappa of .672 for speaker classification (child vs. teacher). Transcription quality is moderate to high with word error rates of .119 for teachers and .238 for children. WSW2.0 exhibits relatively high absolute agreement intraclass correlations (ICC) with expert transcriptions for a range of classroom language features. These include teacher and child mean utterance length, lexical diversity, question asking, and responses to questions and other utterances, which show absolute agreement intraclass correlations between .64 and .98. To establish scalability, we apply the framework to an extensive dataset spanning two years and over 1,592 hours of classroom audio recordings, demonstrating the framework's robustness for broad real-world applications. These findings highlight the potential of deep learning and natural language processing techniques to revolutionize educational research by providing accurate measures of key features of preschool classroom speech, ultimately guiding more effective intervention strategies and supporting early childhood language development.
BibTeX
@inproceedings{sun2025who,
  author = {Anchen Sun and Tiantian Feng and Gabriela Gutierrez and Juan J. Londono and Anfeng Xu and Batya Elbaum and Shrikanth Narayanan and Lynn K. Perry and Daniel S. Messinger},
  title = {{Who Said What (WSW 2.0)? Enhanced Automated Analysis of Preschool Classroom Speech}},
  booktitle = {ICDL},
  year = {2025},
  doi = {10.1109/icdl63968.2025.11204438},
}
Figure