Towards an open-domain chatbot for language practice

Gladys Tyen, Mark Brenchley, Andrew Caines, Paula ButteryNAACL 2022

TL;DR

This paper tackles the challenge of adapting open-domain chatbots for language learning by developing novel decoding strategies to control the difficulty of generated text. They demonstrate that re-ranking candidate responses based on predicted difficulty significantly outperforms simpler vocabulary restriction methods, paving the way for more personalized and effective AI tutors.

What Problem Does This Paper Address?

Learning a new language often requires interactive practice, but traditional methods like one-on-one tutoring can be costly, inaccessible, or even anxiety-inducing for learners. While AI chatbots offer a promising alternative for convenient practice, existing language learning chatbots are often limited to specific contexts and rely on handcrafted rules. This paper addresses the gap of creating open-domain chatbots that can dynamically adjust their language complexity to match a learner's proficiency level, making them more suitable for diverse learners and a wider range of conversational topics.

Methods

The core contribution lies in developing five decoding strategies to control the difficulty of generated text from a pre-trained open-domain chatbot (Facebook's Blender 2.7B).

  • Vocabulary Restriction (Methods 1 & 2): These methods filter out words deemed too difficult for a target CEFR level. Method 1 uses the English Vocabulary Profile (EVP), while Method 2 extends this by training a regressor to predict CEFR levels for words not in the EVP.
  • Re-ranking (Methods 3, 4, & 5): These are more sophisticated approaches that leverage multiple candidate responses generated by the chatbot.
  • Method 3 (Re-ranking): Ranks candidate messages based on a combination of their original probability from the language model and their predicted CEFR difficulty level.
  • Method 4 (Re-ranking with Sub-token Penalties): Enhances Method 3 by introducing penalties for sub-tokens that are too difficult for the target CEFR level, aiming to steer generation towards appropriate complexity.
  • Method 5 (Re-ranking with Sub-token Penalties and Filtering): Builds upon Method 4 by adding a vocabulary filter to remove nonsensical or clearly out-of-vocabulary words, further refining the output quality.

Datasets

  • English Vocabulary Profile (EVP): Used for vocabulary restriction in Method 1.
  • Cambridge Exams (CE) dataset: Used to train the sentence difficulty prediction model for re-ranking methods. This dataset contains English texts from various CEFR levels.
  • Social Security Administration & United States Census Bureau data: Used to supplement vocabulary lists with common names.
  • Exquisite Corpus: Used for word frequency estimation in Method 2.

Key Insights and Ideas

  • Re-ranking is superior to vocabulary restriction: The paper's most significant finding is that re-ranking candidate responses based on predicted difficulty is far more effective than simply filtering out difficult words. This suggests that controlling the overall structure and flow of a sentence is more crucial than just word choice for adjusting difficulty.
  • Sub-token penalties improve difficulty control: Adding penalties for difficult sub-tokens (Method 4) further refines the re-ranking approach, allowing for more nuanced control over text complexity.
  • Filtering is necessary for quality: While re-ranking improves difficulty, adding a vocabulary filter (Method 5) is crucial to mitigate the generation of nonsensical words, a common issue with sub-token-level manipulation.
  • CEFR distribution challenges: The original chatbot's tendency to generate text in the B1-B2 range, even when targeting higher levels, posed a challenge. This highlights that the base model's inherent biases can impact the effectiveness of difficulty adjustment strategies.
  • A1 level is difficult to generate: The study found no dialogues generated at the A1 (beginner) level, suggesting that creating truly simple and engaging content for absolute beginners is a significant challenge.

Key Results

  • Method 5 (Re-ranking with sub-token penalties and filtering) performed best: It achieved the strongest correlation (Spearman's ρ = 0.755, Pearson's r = 0.731) and lowest Mean Absolute Error (MAE = 1.090) between the target and true CEFR levels as judged by human examiners.
  • Vocabulary restriction methods performed poorly: Methods 1 and 2 showed minimal correlation and high MAE, indicating their ineffectiveness in controlling text difficulty.
  • Grammaticality, sensibleness, and specificity remained high: Despite the difficulty adjustment, the quality metrics for grammar, sensibleness, and specificity were statistically equivalent to the original chatbot, suggesting that the proposed methods do not significantly degrade output quality.
  • CEFR distribution: Re-ranking methods produced a more even distribution of generated text across CEFR levels compared to vocabulary restriction methods, which tended to cluster at lower levels.

Limitations and Future Work

  • Focus on self-chat: The evaluation was based on self-chats, which might not perfectly reflect interactive difficulty for a human learner. Future work should directly involve learners.
  • Base model limitations: The inherent difficulty distribution of the base chatbot model (Blender) influenced the results, particularly for higher CEFR levels and the absence of A1 level generation.
  • Quality vs. Difficulty Trade-off: While quality metrics remained statistically similar, there was a slight degradation observed. Future work could explore methods to further improve quality, such as grammatical error correction or style classifiers.
  • Beyond vocabulary: The paper acknowledges that text complexity involves more than just vocabulary, and future work could explore adjusting sentence structure and paraphrasing.
  • Ethical considerations: The paper highlights ethical concerns like toxic language, inaccurate information, and the "human likeness" effect, which need to be addressed in future chatbot development.

Summary updated

BibTeX
@inproceedings{tyen2022towards,
  author = {Gladys Tyen and Mark Brenchley and Andrew Caines and Paula Buttery},
  title = {{Towards an open-domain chatbot for language practice}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}

Loading related content...

Figure