The impact of chatbots based on large language models on second language vocabulary acquisition

Zhihui Zhang, Xiaomeng HuangHeliyon 2024 · DOI

TL;DR

This study investigates the impact of Large Language Model (LLM)-powered chatbots on second language (L2) vocabulary acquisition. The findings demonstrate that using an LLM chatbot significantly improves both receptive and productive vocabulary knowledge, and even aids in long-term retention and incidental learning. This highlights the potential of LLM chatbots as powerful tools for personalized language education.

What Problem Does This Paper Address?

The integration of Artificial Intelligence (AI) into education, particularly for Personalized Language Learning (PLL), has gained considerable traction. While AI chatbots show promise in automating and enhancing educational tasks, their effectiveness in L2 vocabulary acquisition, especially concerning long-term retention and incidental learning, remains an area needing further exploration. Previous research often focused on rule-based chatbots with limitations in natural language understanding and personalized responses. This study aims to bridge this gap by evaluating LLM-based chatbots, which can better mimic human language complexity and offer more tailored support, to understand their impact on L2 vocabulary learning.

Methods

The study employed a quasi-experimental design with 52 foreign language students divided into two groups: an experimental group (EG) using an LLM-powered chatbot for vocabulary learning and a control group (CG) using alternative digital resources. Both groups learned the same vocabulary from the "Cambridge THINK A2" series over eight weeks.

Key methods included:

  • Vocabulary Assessment: Participants underwent immediate post-treatment tests (receptive and productive vocabulary) and delayed tests two weeks later to assess retention.
  • Systematic Observation: The EG's interactions with the chatbot were monitored to analyze frequency of use, number of questions, and types of questions asked.
  • LLM Chatbot Design: The study utilized a custom-built LLM chatbot with a backend service, search service, and AI robot, designed for vocabulary learning, conversation support, and personalized learning. The chatbot's architecture is based on the Transformer model and incorporates Reinforcement Learning from Human Feedback (RLHF) and Proximal Policy Optimization (PPO).

Datasets

The study used the "Cambridge THINK A2" textbook series as the primary source for vocabulary learning materials. The LLM itself was pre-trained on large-scale datasets, as is standard for LLMs, but specific details of these datasets are not provided in the context of this study's evaluation. No new benchmarks were introduced; rather, existing vocabulary tests were adapted and utilized.

Key Insights and Ideas

  • LLMs are Game-Changers for Vocabulary: The study strongly suggests that LLM-based chatbots are not just incremental improvements but significant advancements in L2 vocabulary acquisition tools.
  • Beyond Receptive Learning: A key insight is the chatbot's effectiveness in improving productive vocabulary, which is often more challenging to develop than receptive skills.
  • Incidental Learning Boost: The chatbot facilitates "learning by osmosis," where students pick up new words unintentionally through interaction, a crucial aspect of natural language acquisition.
  • Personalization is Key: The LLM's ability to tailor responses and provide context-specific explanations is a major driver of its success, addressing individual learning needs.
  • Engagement and Reduced Anxiety: The interactive and supportive nature of the chatbot appears to foster greater engagement and potentially reduce the anxiety associated with language learning.

Key Results

  • Immediate Gains: In the immediate post-test, the experimental group (EG) using the chatbot showed significantly higher accuracy scores than the control group (CG) in both productive vocabulary (73.93% vs. 66.24%) and receptive vocabulary (67.34% vs. 64.27%).
  • Sustained Improvement: These gains were maintained in the delayed test, with the EG scoring 67.95% in productive vocabulary and 58.37% in receptive vocabulary, compared to the CG's 55.55% and 53.65%, respectively. The advantage of the EG was more pronounced in the productive test over time.
  • High Engagement: Students in the EG used the chatbot frequently (over five times per week, with 76.92% using it more than seven times per week) and asked a substantial number of questions (average of 24.37 per week).
  • Incidental Learning: Over 69% of students in the EG learned more than 10 non-target vocabulary words, indicating effective incidental learning.
  • Question Types: The most common questions were about word meaning (39.29%), followed by usage examples (22.41%), synonyms/antonyms (21.31%), and contextual information (16.98%).

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Limited Sample Size: The study involved 52 participants, and further research with larger and more diverse samples is warranted.
  • Focus on Vocabulary: While effective for vocabulary, future studies should explore the impact of LLM chatbots on other aspects of language learning, such as grammar, pronunciation, and overall fluency.
  • Student Interest and Motivation: Further investigation is needed into how LLM chatbots influence student interest, motivation, and self-directed learning.
  • Underlying Factors: Understanding the factors that influence student engagement and participation with chatbots is crucial.
  • Cooperative Learning: Exploring how LLM chatbots can foster cooperation and sharing among learners is a promising avenue.
  • Higher-Order Thinking: Investigating the potential of LLM chatbots to enhance higher-order thinking skills, such as reasoning, is also suggested.
  • Chatbot Diversity: The study emphasizes that different types of chatbots may yield varying effects, highlighting the need for research on diverse chatbot models.

Summary updated

Abstract

In recent years, the integration of artificial intelligence (AI) and machine learning (ML) into education, particularly for Personalized Language Learning (PLL), has garnered significant attention. This approach tailors interventions to address the unique challenges faced by individual learners. Large Language Models (LLMs), including Chatbots, have demonstrated a substantial potential in automating and enhancing educational tasks, effectively capturing the complexity and diversity of human language. In this study, 52 foreign language students were randomly divided into two groups: one with the assistance of a Chatbot based on LLMs and one without. Both groups learned the same series of target words over eight weeks. Post-treatment assessments, including systematic observation and quantitative tests assessing both receptive and productive vocabulary knowledge, were conducted immediately after the study and again two weeks later. The findings demonstrate that employing an AI Chatbot based on LLMs significantly aids students in acquiring both receptive and productive vocabulary knowledge during their second language learning journey. Notably, Chatbots contribute to the long-term retention of productive vocabulary and facilitate incidental vocabulary learning. This study offers valuable insights into the practical benefits of LLM-based tools in language learning, with a specific emphasis on vocabulary development. Chatbots utilizing LLMs emerge as effective language learning aids. It emphasizes the importance of educators understanding the potential of these technologies in L2 vocabulary instruction and encourages the adoption of strategic teaching methods incorporating such tools.
BibTeX
@inproceedings{zhang2024impact,
  author = {Zhihui Zhang and Xiaomeng Huang},
  title = {{The impact of chatbots based on large language models on second language vocabulary acquisition}},
  booktitle = {Heliyon},
  year = {2024},
  doi = {10.1016/j.heliyon.2024.e25370},
}
Figure