Curriculum-Driven Edubot: A Framework for Developing Language Learning Chatbots through Synthesizing Conversational Data

Yu Li, Shang Qu, Jili Shen, Shangchao Min, Zhou YuSigdial 2024 · DOI

TL;DR

This paper introduces Curriculum-Driven EduBot, a novel framework for creating language learning chatbots that are deeply integrated with educational curricula. By synthesizing textbook content and conversational data, EduBot offers a more engaging and effective learning experience than general-purpose chatbots like ChatGPT, as demonstrated by user studies.

What Problem Does This Paper Address?

Traditional language learning often follows a structured, textbook-based curriculum. However, existing conversational AI agents, while useful, typically lack this structured approach. They often engage in generic conversations that may not align with a student's learning objectives or proficiency level, potentially hindering progress. The challenge is to bridge the gap between the dynamic, interactive nature of chatbots and the systematic, curriculum-driven approach of traditional language education.

Methods

The core of this work is a framework for developing Curriculum-Driven EduBot. The process involves several key steps:

  1. Topic Augmentation: Relevant topics are extracted from a chosen English textbook. Large Language Models (LLMs), specifically ChatGPT, are then used to generate a broader set of related subtopics to enrich the conversational scope.
  2. Persona Synthesis: Fixed-format personas are created for two dialogue participants. One persona is consistently set as a student, while the other is randomized to create diverse conversational partners. This "fixed-random" strategy aims to make the chatbot anticipate a student user and generate more relatable conversations.
  3. Dialogue Synthesis: Using the augmented topics and synthesized personas, LLMs generate synthetic dialogues. Crucially, these dialogues incorporate vocabulary from the textbook to ensure curriculum alignment and appropriate language difficulty. The chatbot persona is prompted to actively lead the conversation, guiding the student.
  4. Fine-tuning an Open-Source Model: The synthesized conversational data is used to fine-tune an open-source LLM (Vicuna-13B). This fine-tuning process imbues the model with the curriculum-specific knowledge, personas, and conversational style, creating the EduBot.

Datasets

  • Training Data: The authors synthesized a large dataset of dialogues (880-1,210 dialogues per unit, averaging 1,058.76 dialogues) based on the "New College English" (3rd edition) textbook. Each dialogue averaged 11.77 utterances and 28.71 words.
  • Evaluation Data: User studies were conducted with 24 Chinese college students who had recently taken "College English 4." Participants engaged in conversations with both EduBot and ChatGPT.

Key Insights and Ideas

  • Curriculum Integration is Key: The most significant insight is that aligning chatbot conversations with a specific curriculum dramatically improves their effectiveness for language learning. This goes beyond simply using textbook topics; it involves synthesizing dialogues that mimic the structure and vocabulary of the learning material.
  • Persona-Driven Conversations Enhance Realism: The use of synthesized personas, especially the "fixed-random" approach, leads to more natural and engaging interactions. This helps the chatbot feel more like a conversational partner rather than just an information provider.
  • Active Guidance is Crucial: Prompting the chatbot persona to lead the conversation by asking questions proved effective in keeping users engaged and encouraging them to practice their own conversational skills.
  • CEFR Alignment for Proficiency: Explicitly controlling the English proficiency level of synthesized dialogues using the CEFR scale is vital for matching the chatbot's output to the student's learning level.

Key Results

  • EduBot Outperforms ChatGPT: User studies showed that EduBot was preferred over ChatGPT in several key areas.
  • Topic Relevance: 50.0% of participants found EduBot's topics more relevant to the course compared to 16.7% for ChatGPT.
  • Naturalness and Realism: 62.5% found EduBot's interactions natural and realistic, versus 4.2% for ChatGPT.
  • Guidance: 75% of participants felt EduBot actively guided the conversation with questions, compared to 4.2% for ChatGPT.
  • Vocabulary Understanding: 37.5% found ChatGPT used vocabulary they didn't understand, compared to 20.8% for EduBot.
  • High Recommendation Rate: 87.5% of students believed EduBot could help them improve their conversational skills, and 83.3% were willing to recommend it.
  • Utterance Length: ChatGPT produced longer utterances on average (approx. 10 words more than EduBot), sometimes exceeding 60 words, which is less natural for daily conversations. EduBot's shorter, more interactive utterances encouraged longer user responses.
  • Target Vocabulary Usage: Conversations with EduBot included an average of 5.55 target vocabulary words, significantly more than ChatGPT's 0.62.

Limitations and Future Work

  • Limitations:
    • EduBot occasionally included descriptions of emotions or actions that were unnatural for an AI.
  • It sometimes made incorrect assumptions about the user's feelings or conversation context (hallucinations). These issues are attributed to the data used for fine-tuning.
  • Future Work:
    • Refine data synthesis processes and implement stricter post-processing to filter out unnatural content.
  • Expand the content spectrum.
  • Incorporate multimedia elements.
  • Introduce real-time feedback mechanisms.
  • The authors hope EduBot will evolve into an indispensable learning companion.

Summary updated

BibTeX
@inproceedings{li2024curriculum,
  author = {Yu Li and Shang Qu and Jili Shen and Shangchao Min and Zhou Yu},
  title = {{Curriculum-Driven Edubot: A Framework for Developing Language Learning Chatbots through Synthesizing Conversational Data}},
  booktitle = {Sigdial},
  year = {2024},
  doi = {10.18653/v1/2024.sigdial-1.35},
}
Figure