The ADAIO System at the BEA-2023 Shared Task: Shared Task Generating AI Teacher Responses in Educational Dialogues

Adaeze Adigwe, Zheng YuanACL 2023

TL;DR

This paper introduces the ADAIO system, which achieved second place in the BEA-2023 Shared Task on generating AI teacher responses. The system leverages OpenAI's GPT-3 model with a few-shot prompting approach, demonstrating the effectiveness of large language models in simulating human-like teacher interactions for educational dialogues.

What Problem Does This Paper Address?

The paper tackles the challenge of creating AI systems that can effectively act as "AI teachers" in educational dialogues. The goal is to develop Intelligent Tutoring Systems (ITS) that not only provide factually accurate information but also engage students in a teacher-like manner, understand their needs, and help them improve their comprehension. This requires generating responses that are natural, contextually relevant, and pedagogically sound, which is a significant hurdle for current AI models, especially in capturing nuanced teaching strategies and long-range conversational dependencies.

Methods

The ADAIO system's core is built upon OpenAI's GPT-3 large language models (LLMs). The researchers experimented with several approaches:

  • Model Selection: They found that the text-davinci-003 model produced the most human-like and contextually relevant responses compared to other GPT-3 variants like ada, curie, and babbage.
  • Prompt Engineering: A key innovation was the meticulous design of various prompts (Prompts A-E) to guide GPT-3's response generation. These prompts explored different aspects of the AI teacher's role, including politeness, helpfulness, probing student understanding, and encouraging conversational skills.
  • Training Methods:
    • Zero-Shot: GPT-3 was given a prompt without any examples.
  • Few-Shot: This involved providing GPT-3 with the prompt along with five handpicked example dialogues from the training set that demonstrated a teaching focus. This proved to be the most effective method.
  • Fine-tuning: The text-ada-001 model was fine-tuned on the TSCC corpus.

The final system utilized the text-davinci-003 model with a few-shot prompt-based approach (specifically Prompt A).

Datasets

The primary dataset used was the Teacher-Student Chatroom Corpus (TSCC). This corpus consists of 102 chatrooms where English as a Second Language (ESL) teachers interact with students. The dialogues are used to assess language proficiency and focus on everyday speech and language usage. The dataset was split into training, development, and test sets. The researchers also used a subset of the training data for internal model selection.

Key Insights and Ideas

  • Prompting is Powerful: The study highlights that carefully crafted prompts can significantly influence the tone, style, and effectiveness of LLM-generated responses, allowing them to embody specific pedagogical roles.
  • Few-Shot Learning Shines: The few-shot learning approach, where the model is given a few examples, was crucial for achieving high performance, demonstrating LLMs' ability to learn from context and adapt to specific tasks like AI teaching.
  • GPT-3's Adaptability: GPT-3, particularly text-davinci-003, showed remarkable ability to handle educational dialogues, even those containing grammatical errors or non-canonical language, without explicit error correction.
  • Multilingual Potential: The dialogue-based nature of the TSCC corpus and the multilingual capabilities of LLMs suggest a promising avenue for addressing language barriers in educational applications.

Key Results

The ADAIO system, using the few-shot text-davinci-003 model with Prompt A, achieved second place in the BEA-2023 Shared Task. While machine evaluation metrics like BERTScore showed minor differences across models, the qualitative performance of the few-shot davinci-003 model was deemed superior for the task's requirements. The DialogRPT metric also supported the system's strong performance, with the ADAIO system ranking second overall.

Limitations and Future Work

  • Reliance on Automatic Metrics: The authors acknowledge a limitation in relying solely on automatic evaluation metrics and suggest that human evaluation is crucial for a comprehensive assessment of AI teacher quality.
  • Prompt Engineering Complexity: While effective, prompt engineering requires significant experimentation and expertise. Future work could explore more automated methods for prompt discovery.
  • Pedagogical Depth: The paper suggests further research into adapting ITS to specific teaching instruction methods and pedagogies, such as constructivist learning or productive failure, to enhance their educational efficacy.
  • Multilingual Applications: The authors advocate for the development and adoption of open-source multilingual LLMs to facilitate their use in diverse linguistic contexts and for L2 acquisition.

Summary updated

BibTeX
@inproceedings{adigwe2023adaio,
  author = {Adaeze Adigwe and Zheng Yuan},
  title = {{The ADAIO System at the BEA-2023 Shared Task: Shared Task Generating AI Teacher Responses in Educational Dialogues}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure