Does a code-switching dialogue system help users learn conversational fluency in Choctaw?

Jacqueline Brixey, David TraumNAACL 2025

TL;DR

This paper explores whether a code-switching chatbot can improve conversational fluency in Choctaw, an endangered language, compared to a monolingual version. While both chatbots facilitated learning, the code-switching version led to a slightly better user experience and higher engagement, suggesting potential for maintaining long-term motivation in language learning.

What Problem Does This Paper Address?

Learning endangered languages, particularly Indigenous languages in North America like Choctaw, presents significant challenges. Access to fluent speakers is often limited due to geographical dispersion and a declining number of speakers. This scarcity of practice opportunities hinders the development of conversational fluency, a key goal for language learners. Traditional language learning environments can also be intimidating, with learners fearing mistakes or judgment. The authors propose that chatbots can serve as valuable, non-judgmental conversational partners, especially for languages with limited resources. This research specifically investigates whether incorporating code-switching (alternating between Choctaw and English) in a chatbot can enhance the learning experience and outcomes for Choctaw learners, building on previous work with a similar bilingual chatbot.

Methods

The researchers implemented two chatbots: a monolingual Choctaw version and an English-Choctaw code-switching version. Both chatbots were built using NPCEditor, a response classifier and dialogue management system trained on a question-answer corpus. The core of the system involved handcrafted utterances and a statistical classifier to select appropriate responses.

For the code-switching chatbot, they employed translanguaging techniques, incorporating English explanations for key Choctaw vocabulary within the dialogue. This approach aimed to mimic natural bilingual communication patterns. The monolingual chatbot, conversely, was designed to align with an immersion-style pedagogy, often discouraging or ignoring non-target language use.

To evaluate the systems, participants underwent a pre- and post-interaction language test assessing vocabulary and syntax gains. They also completed a post-interaction survey to gauge their experience, including rapport, naturalness, and connection to the language.

Datasets

The primary dataset used for the chatbot's domain knowledge consisted of ten animal stories from the ChoCo (Choctaw language corpus). These stories, originally in Choctaw, were provided with English translations. The chatbot's responses were handcrafted based on these stories. The QA corpus used for training NPCEditor was also handcrafted.

Key Insights and Ideas

  • Code-switching as a confidence booster: The study suggests that code-switching might make learners feel more comfortable and less threatened, potentially leading to increased confidence and a willingness to attempt more vocabulary questions. This aligns with theories of "face" in communication, where learners feel more accepted when their full linguistic repertoire is acknowledged.
  • Translanguaging's role in engagement: The code-switching chatbot, designed with translanguaging principles, fostered a more engaging and enjoyable user experience. This indicates that allowing learners to leverage their existing language knowledge can positively impact their motivation and perception of the learning tool.
  • No significant learning advantage for code-switching: Despite the improved user experience, the language tests did not show a statistically significant difference in learning gains between the monolingual and code-switching chatbots. This suggests that while code-switching might enhance the experience of learning, the core learning of vocabulary and syntax can occur with either approach.
  • Chatbots as valuable tools for endangered languages: The research reinforces the idea that chatbots can be effective tools for practicing languages with limited speaker availability, offering a safe and accessible space for learners to engage with the language.

Key Results

  • User Experience: The code-switching group reported a significantly friendlier chatbot and expressed a greater likelihood of recommending the system to others. They also found the interaction more interesting and natural.
  • Language Learning: Both groups showed learning gains in vocabulary and grammar. The monolingual group showed a slightly higher average improvement in correct vocabulary answers (1.5 vs. 1.36) and grammar questions attempted (0.1 vs. 0.09), but these differences were not statistically significant.
  • Engagement: The code-switching group scored higher on measures of engagement and connection, indicating they felt more connected to the system and more willing to continue the conversation.

Limitations and Future Work

  • Short-term interaction: The study involved a single 15-minute interaction. Future work could evaluate learning gains over longer periods or multiple sessions to assess retention and sustained engagement.
  • Computational limitations: The system had difficulty processing some unique characters in participants' input, requiring them to use alternative spellings, which may have impacted user satisfaction.
  • Generalizability: The findings are specific to Choctaw. Replication with other language communities is needed to confirm if these results are generalizable.
  • Data processing: The system's inability to process certain characters and potential rendering issues are noted as computational limitations.
  • Future directions: The authors suggest exploring longer-term interactions, investigating retention, and replicating the study with other language communities. They also highlight the potential of the collected conversation logs as a valuable resource for language instructors and linguists.

Summary updated

BibTeX
@inproceedings{brixey2025does,
  author = {Jacqueline Brixey and David Traum},
  title = {{Does a code-switching dialogue system help users learn conversational fluency in Choctaw?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure