Automating Analysis and Feedback to Improve Mathematics Teachers' Classroom Discourse

Abhijit Suresh, Tamara Sumner, Jennifer Jacobs, Bill Foland, Wayne H. WardAAAI 2019 · DOI

Introduction

In high-quality mathematics education, teachers act as facilitators of "accountable talk," utilizing specific discursive strategies known as talk moves to foster student reasoning. However, providing teachers with detailed feedback on these strategies currently requires manual coding of transcripts by experts—a process that is expensive and difficult to scale. Researchers at the University of Colorado Boulder have developed a deep learning model to automate this analysis, achieving a 65% F1 score and paving the way for "TalkBack," an application for personalized teacher feedback.

Methodology

The core of the system is a multi-class classification model based on a Bidirectional Long Short-Term Memory (bi-LSTM) network. The model processes data organized into "turns"—a student utterance followed by a teacher utterance—to maintain contextual awareness.

The proposed model architecture, illustrating the flow from transcript sentences to talk move classification.

The input representation combines four distinct feature types:

  1. Sentence Embeddings: 600-dimensional vectors capturing semantic meaning.
  2. GloVe Bag-of-Words (BOW): 300-dimensional vectors for average word meaning.
  3. CountVectorizer: 100-dimensional vectors tracking high-frequency keywords.
  4. Role: A binary feature identifying the speaker (teacher vs. student).

The architecture includes a bi-LSTM layer followed by a time-distributed dense layer and two additional dense layers with ReLU activation, ending in a 7-way softmax output.

Dataset and Labeling

The study utilized 406 lesson transcripts comprising 100,683 sentences. Human annotators coded teacher sentences into six mutually exclusive talk moves:

  1. Keeping everyone together
  2. Getting students to relate to another’s ideas
  3. Restating
  4. Revoicing
  5. Pressing for accuracy
  6. Pressing for reasoning

The dataset showed high inter-rater agreement (90%), though it was naturally skewed toward certain moves like "Pressing for Accuracy."

Key Results and Insights

The bi-LSTM model outperformed other recurrent architectures, achieving an overall F1 score of 0.646 (approximately 65%). This represents a significant improvement over the majority-class baseline.

Comparison of F1 scores across different recurrent layer types, with Bi-LSTM showing the highest performance.

The model was particularly effective at identifying "Pressing for Accuracy" (F1 = 0.74) and "Pressing for Reasoning" (F1 = 0.71). A key insight was the importance of context; including student sentences allowed the model to better distinguish teacher intent. Additionally, the inclusion of simple word counts (CountVectorizer) provided a 3% performance boost, suggesting that specific keywords are highly indicative of certain moves.

Limitations and Future Directions

While successful, the model struggled with "Revoicing" (F1 = 0.38), indicating a need for deeper semantic understanding of paraphrasing. Future work aims to expand classification to student discourse moves and integrate the model into the "TalkBack" web application. This tool will provide teachers with automated dashboards to visualize and improve their classroom discourse patterns over time.

Summary updated

BibTeX
@inproceedings{suresh2019automating,
  author = {Abhijit Suresh and Tamara Sumner and Jennifer Jacobs and Bill Foland and Wayne H. Ward},
  title = {{Automating Analysis and Feedback to Improve Mathematics Teachers' Classroom Discourse}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019721},
}
Figure