The Teacher-Student Chatroom Corpus version 2: more lessons, new annotation, automatic detection of sequence shifts

Andrew Caines, Helen Yannakoudakis, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, Paula ButteryLinköping 2022 · DOI

Abstract

The second version of the Teacher-Student Chatroom Corpus (TSCC) significantly expands upon its predecessor, offering a rich dataset for analyzing one-to-one English language tutoring. TSCC v2 includes 260 chatroom lessons, totaling 41.4K conversational turns and 363K word tokens across 13 students of varying proficiency levels (B1–C2). Beyond data expansion, this release introduces pedagogical annotations using the Self-Evaluation of Teacher Talk (SETT) framework and explores the use of transformer-based models to automatically detect discourse sequence shifts. The corpus is freely available for research, providing a foundation for developing pedagogically aware AI tutors and enhancing teacher training.

Introduction: The Challenge of One-to-One Tutoring Data

While massive datasets exist for generic web-based chat, high-quality data specifically capturing the nuances of language tutoring remains scarce. In a pedagogical setting, dialogue is structured into "sequences"—functional blocks moving from openings and warm-ups to grammar exercises or "repair" sequences following student errors. Understanding these transitions is crucial for helping teachers reflect on their interactional competence and for building AI dialogue managers that can steer lessons naturally rather than just responding to prompts in isolation.

The TSCC v2 Dataset

The primary contribution of this work is the significant expansion of the TSCC. Version 2 increases the number of lessons from 102 to 260, involving two teachers and 13 students with seven different first languages (Italian, Japanese, Mandarin, Russian, Spanish, Thai, and Ukrainian).

Comparison of TSCC Version 1 and Version 2 metrics showing growth in lessons, turns, and word counts.

The corpus now covers a broader range of learner proficiencies, from intermediate (B1) to proficient (C2) on the CEFR scale. This volume and diversity make it a robust resource for studying how teachers manage one-to-one interactions in a digital environment.

Pedagogical Annotation and Sequence Detection

To enrich the data, the researchers applied the Self-Evaluation of Teacher Talk (SETT) framework. This categorizes turns into "modes" (e.g., Managerial, Materials/Skills) and "interactures" (e.g., Scaffolding, Direct Repair), providing a granular view of how a lesson is actually constructed turn-by-turn.

Example of an annotated chat transcript showing sequence labels like 'opening', 'repair', and 'scaffolding' alongside corrected text.

A key finding from this annotation is the teacher's role as the primary architect of the lesson. While conversational turns are split nearly 50/50, teachers set three times as many "modes" as students, focusing heavily on managerial tasks.

The researchers also framed lesson management as a machine learning task: New Sequence Detection. Using DistilBERT, they experimented with predicting whether the next turn would initiate a new discourse sequence. They found that providing the model with conversational context (the previous turn) significantly improved performance, suggesting that topic shifts are often foreshadowed in the preceding exchange.

Key Results and Future Directions

The experiments demonstrated that detecting sequence shifts is a solvable task. The best-performing configuration—DistilBERT with one turn of context—achieved an F1-score of 0.489, significantly outperforming probabilistic baselines. Notably, the model achieved a high recall (0.636), which is particularly promising for real-time applications where missing a topic shift is more disruptive than a false alarm.

Future work will address the subjectivity of sequence boundaries and move toward multi-class classification to predict the specific type of upcoming sequence (e.g., "Grammar Repair" vs. "Homework Setting"). The ultimate goal is to integrate these models into real-time feedback tools for teachers and more sophisticated, pedagogically aware language learning chatbots.

Summary updated

Abstract

The first version of the Teacher-Student Chatroom Corpus (TSCC) was released in 2020 and contained 102 chatroom dialogues between 2 teachers and 8 learners of English, amounting to 13.5K conversational turns and 133K word tokens. In this second version of the corpus, we release an additional 158 chatroom dialogues, amounting to an extra 27.9K conversational turns and 230K word tokens. In total there are now 260 chatroom lessons, 41.4K conversational turns and 363K word tokens, involving 2 teachers and 13 students with seven different first languages. The content of the lessons was, as before, guided by the teacher, and the proficiency level of the learners is judged to range from B1 to C2 on the CEFR scale. Annotation of the dialogue continued with conversational analysis of sequence types, pedagogical focus, and correction of grammatical errors. In addition, we have annotated fifty of the dialogues using the Self-Evaluation of Teacher Talk framework which is intended for self-reflection on interactional aspects of language teaching. Finally, we conducted machine learning experiments to automatically detect shifts in discourse sequences from turn to turn, using modern transfer learning methods with large pretrained language models. The TSCC v2 is freely available for research use.
BibTeX
@inproceedings{caines2022teacher,
  author = {Andrew Caines and Helen Yannakoudakis and Helen Allen and Pascual Pérez-Paredes and Bill Byrne and Paula Buttery},
  title = {{The Teacher-Student Chatroom Corpus version 2: more lessons, new annotation, automatic detection of sequence shifts}},
  booktitle = {Linköping},
  year = {2022},
  doi = {10.3384/ecp190003},
}
Figure