ChatBack: Investigating Methods of Providing Grammatical Error Feedback in a GUI-based Language Learning Chatbot

Kai-Hui Liang, Sam Davidson, Xun Yuan, Shehan Panditharatne, Chun-Yen Chen, Ryan Shea, Derek Pham, Yinghua Tan, Erik Voss, Luke Fryer, Zhou YuACL 2023

TL;DR

This paper explores how to best deliver grammatical error feedback in AI-powered language learning chatbots. They found that feedback delivered via a separate GUI interface, rather than directly from the chatbot, leads to less user frustration. Furthermore, guided self-correction, where learners are prompted to fix their own errors, is more beneficial for learning and confidence, especially for less proficient learners.

What Problem Does This Paper Address?

Learning a new language often requires extensive practice, and AI chatbots are emerging as valuable tools for this. A crucial aspect of language learning is receiving effective feedback on grammatical errors. While synchronous feedback (given in real-time) is generally more effective than asynchronous feedback, and self-correction can be more beneficial than explicit correction, current language learning chatbots often lack robust synchronous feedback and self-correction capabilities. This paper aims to investigate optimal strategies for delivering this feedback within a chatbot interface, specifically focusing on how feedback is presented (e.g., directly from the chatbot vs. a separate GUI) and the process of correction (guided self-correction vs. explicit correction).

Methods

The study proposes a synchronous conversational corrective feedback (CCF) method integrated into a GUI-based chatbot. This system uses a grammar error correction (GEC) model (a fine-tuned T5 model) to identify errors and then presents feedback through different strategies. The core novelty lies in comparing two main feedback delivery methods:

  1. Feedback from the chatbot: The AI conversational partner directly provides the feedback.
  2. Feedback from a GUI: Feedback is presented in a separate graphical user interface window, distinct from the conversational flow.

Additionally, the study compares two correction processes:

  1. Guided self-correction: Learners are given hints and opportunities to correct their own errors, with increasing levels of support.
  2. Explicit correction: The system directly provides the correct form without requiring the learner to self-correct.

The study involved 175 native Mandarin speakers practicing English through conversations with the chatbot. Their experiences were evaluated through post-conversation surveys measuring frustration, self-efficacy, perceived usefulness, enjoyment, and intention to use the system.

Datasets

  • ErAConD dataset: Used for fine-tuning the T5 grammar error correction (GEC) model. This dataset consists of grammatical error annotated conversational dialogues between L2 English learners and an educational chatbot.
  • No new benchmark datasets were introduced.

Key Insights and Ideas

  • Separation is key for reducing frustration: Users experienced significantly less frustration when feedback was delivered via a GUI rather than directly from the chatbot. This suggests that interrupting the conversational flow with direct corrections can be off-putting, and a separate interface offers a less intrusive experience.
  • Guided self-correction empowers learners: Providing learners with opportunities to self-correct, with graduated levels of support, led to increased self-efficacy and perceived usefulness of the feedback. This active participation seems to foster a deeper understanding and confidence.
  • Learner characteristics matter: The effectiveness of feedback strategies is not one-size-fits-all.
  • Higher lexical diversity: Users with higher lexical diversity found guided self-correction more annoying, possibly because they already grasp the rules and find the process redundant.
  • Serious learners: Those with a strong motivation to improve their English (defined as "serious learners") benefited more from guided self-correction, reporting higher confidence, enjoyment, and perceived usefulness. This highlights the importance of tailoring feedback to learner motivation.

Key Results

  • Reduced Frustration: Users interacting with feedback from the GUI (Group 2) reported significantly lower frustration than those receiving feedback directly from the chatbot (Group 1).
  • Increased Self-Efficacy: Users who received guided self-correction (Group 2) showed greater increases in self-efficacy compared to those receiving explicit correction without self-correction (Group 3).
  • Motivation Matters: Serious learners reported higher enjoyment, self-efficacy, and perceived usefulness with guided self-correction compared to explicit correction.
  • Lexical Diversity Impact: Learners with higher lexical diversity found guided self-correction more annoying than explicit correction.

Limitations and Future Work

  • Linguistic Ability Assessment: The assessment of learners' linguistic ability was limited to a single, short conversation and used a tool primarily designed for writing. Future work should incorporate more comprehensive assessments of receptive and productive skills.
  • Participant Selection: The study excluded participants who couldn't maintain a minimum conversation length, potentially missing insights from learners with very limited proficiency who might benefit differently from feedback.
  • GEC Model Performance: The GEC model used was developed specifically for conversational errors and has room for improvement. Future work will focus on collecting more diverse conversational GEC datasets to enhance model accuracy and effectiveness.
  • Feedback Interruptiveness: All feedback strategies were somewhat interruptive. Future systems could explore non-intrusive methods like on-demand metalinguistic explanations.
  • Personalization: The authors emphasize the need for personalized feedback mechanisms that cater to individual differences in proficiency, motivation, and learning styles.

Summary updated

BibTeX
@inproceedings{liang2023chatback,
  author = {Kai-Hui Liang and Sam Davidson and Xun Yuan and Shehan Panditharatne and Chun-Yen Chen and Ryan Shea and Derek Pham and Yinghua Tan and Erik Voss and Luke Fryer and Zhou Yu},
  title = {{ChatBack: Investigating Methods of Providing Grammatical Error Feedback in a GUI-based Language Learning Chatbot}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure