Impact of Experiencing Misrecognition by Teachable Agents on Learning and Rapport

Yuya Asano, Diane J. Litman, Mingzhi Yu, Nikki G. Lobczowski, Timothy Nokes-Malach, Adriana Kovashka, Erin Walker

Introduction

"Learning by teaching" is a powerful educational strategy where students solidify their own understanding by explaining concepts to others. In educational technology, this has inspired the creation of teachable agents—virtual characters or robots that students "teach" to solve problems. While speech-based interaction offers a natural interface for these agents, it is frequently plagued by Automatic Speech Recognition (ASR) errors.

This paper investigates whether these technical glitches undermine the educational value of the interaction. Specifically, the authors explore how dialogue misrecognitions—errors that cause the agent to respond differently than intended—affect a student's learning gains and their sense of rapport with the agent.

Methodology and Error Categorization

The researchers analyzed data from 40 undergraduate students teaching ratio word problems to a robot named Emma. To move beyond simple Word Error Rates (WER), which often fail to capture the functional impact of an error, the team used a simulation approach. They compared the response Emma actually gave (based on ASR) to the response she would have given if she had received a perfect human transcription of the student's input.

From this comparison, they defined three distinct types of dialogue misrecognition:

  • Prevented: The error caused Emma to give a generic "I don't understand" response, halting progress.
  • Different: Emma suggested a different problem-solving path than the one the student intended.
  • Proceeded: A "happy accident" where Emma moved to the next step despite a misrecognition.
Table showing examples of student utterances, ASR output, and the resulting dialogue categorization.

Key Results

The study found that students are remarkably resilient to conversational noise. Despite an average Word Error Rate of .226, educational outcomes remained stable. Students improved their scores from an average of 5.58 on the pre-test to 6.82 on the post-test, regardless of the frequency of ASR errors encountered.

Statistical analysis revealed no significant correlation between overall misrecognition and either learning gains ($r = .246$) or rapport ($r = -0.016$). However, the type of error mattered. "Prevented" errors (where the robot explicitly stated it did not understand) showed a marginal negative correlation with rapport ($p = .051$), suggesting that social bonds are more sensitive to perceived incompetence than to conversational detours.

Correlations between different types of misrecognition and the primary outcomes of learning gains and rapport.

Discussion and Design Implications

The findings suggest a "Presume Reasonable" design policy for educational robots. Rather than having an agent stop the flow to ask for clarification when it is unsure, it may be more effective for the agent to simply guess and move forward. Students appear capable of filtering out a robot's confusion and staying focused on the underlying material, provided the social interaction isn't repeatedly stalled by "I don't get it" responses.

Limitations and Future Work

While promising, the study is limited by a small sample size ($N=38$ for final analysis) and a potential ceiling effect, as some students scored perfectly on the pre-test. Furthermore, the system utilized legacy AIML pattern-matching. Future research should investigate whether modern Large Language Models (LLMs), which are more "intelligent" but still prone to hallucinations and errors, produce similar levels of resilience in learners. Additionally, future work must address how ASR errors disproportionately affect students with diverse accents or dialects to ensure equitable AI-driven education.

Summary updated

BibTeX
@inproceedings{asano2023impact,
  author = {Yuya Asano and Diane J. Litman and Mingzhi Yu and Nikki G. Lobczowski and Timothy Nokes-Malach and Adriana Kovashka and Erin Walker},
  title = {{Impact of Experiencing Misrecognition by Teachable Agents on Learning and Rapport}},
  booktitle = {AIED (Posters/Late Breaking Results/...)},
  year = {2023},
  doi = {10.1007/978-3-031-36336-8\_94},
}
Figure