Introduction
In high-quality mathematics education, teachers act as facilitators of "accountable talk," utilizing specific discursive strategies known as talk moves to foster student reasoning. However, providing teachers with detailed feedback on these strategies currently requires manual coding of transcripts by experts—a process that is expensive and difficult to scale. Researchers at the University of Colorado Boulder have developed a deep learning model to automate this analysis, achieving a 65% F1 score and paving the way for "TalkBack," an application for personalized teacher feedback.
Methodology
The core of the system is a multi-class classification model based on a Bidirectional Long Short-Term Memory (bi-LSTM) network. The model processes data organized into "turns"—a student utterance followed by a teacher utterance—to maintain contextual awareness.
The input representation combines four distinct feature types:
- Sentence Embeddings: 600-dimensional vectors capturing semantic meaning.
- GloVe Bag-of-Words (BOW): 300-dimensional vectors for average word meaning.
- CountVectorizer: 100-dimensional vectors tracking high-frequency keywords.
- Role: A binary feature identifying the speaker (teacher vs. student).
The architecture includes a bi-LSTM layer followed by a time-distributed dense layer and two additional dense layers with ReLU activation, ending in a 7-way softmax output.
Dataset and Labeling
The study utilized 406 lesson transcripts comprising 100,683 sentences. Human annotators coded teacher sentences into six mutually exclusive talk moves:
- Keeping everyone together
- Getting students to relate to another’s ideas
- Restating
- Revoicing
- Pressing for accuracy
- Pressing for reasoning
The dataset showed high inter-rater agreement (90%), though it was naturally skewed toward certain moves like "Pressing for Accuracy."
Key Results and Insights
The bi-LSTM model outperformed other recurrent architectures, achieving an overall F1 score of 0.646 (approximately 65%). This represents a significant improvement over the majority-class baseline.
The model was particularly effective at identifying "Pressing for Accuracy" (F1 = 0.74) and "Pressing for Reasoning" (F1 = 0.71). A key insight was the importance of context; including student sentences allowed the model to better distinguish teacher intent. Additionally, the inclusion of simple word counts (CountVectorizer) provided a 3% performance boost, suggesting that specific keywords are highly indicative of certain moves.
Limitations and Future Directions
While successful, the model struggled with "Revoicing" (F1 = 0.38), indicating a need for deeper semantic understanding of paraphrasing. Future work aims to expand classification to student discourse moves and integrate the model into the "TalkBack" web application. This tool will provide teachers with automated dashboards to visualize and improve their classroom discourse patterns over time.
Summary updated