Beyond Bilingual: Studying Quadrilingual Code-Switching with Conversational AI

Annika Bush, Melika Minnaar, Salys Sultan, Tasnimul Hasan Tashin, Ankur Bhatt, Meltem AksoyarXiv 2026 · arXiv

TL;DR

This study investigates how quadrilingual speakers in Aruba and Curaçao navigate interactions with Large Language Models (LLMs) across four languages: English, Dutch, Spanish, and the low-resource Creole Papiamentu/o. The researchers found that while English remains the "default" for AI interaction, users naturally switch languages based on the conversation topic—moving toward local languages for cultural discussions—and generally feel as comfortable switching with a chatbot as they do with a human.

What Problem Does This Paper Address?

Most research on code-switching—the practice of alternating between two or more languages in a single conversation—focuses on bilingual pairs of high-resource languages (like English-Spanish). However, millions of people live in truly multilingual environments where they juggle three, four, or more languages daily.

The authors tackle a critical gap: how do these speakers interact with AI agents that support their languages unequally? In the Caribbean islands of Aruba and Curaçao, residents are predominantly quadrilingual. While English and Spanish are "high-resource" (well-supported by AI), Dutch is mid-resource, and Papiamentu/o is a "low-resource" Creole that has only recently been added to major translation systems. The paper explores whether users "narrow" their linguistic repertoire to accommodate the AI or if they bring their full, fluid multilingual identity to the chat.

Methods

The researchers conducted a preregistered interaction study with 42 university students from Aruba and Curaçao, plus a comparison group of 20 international students.

The study procedure followed a structured workflow:

  1. Language Selection: Participants chose a preferred language for the survey interface.
  2. Chat 1 (Institutional): A 5-minute conversation about university life and career goals.
  3. Survey 1: Assessing AI familiarity and attitudes.
  4. Chat 2 (Cultural): A 5-minute conversation about local food, traditions, and culture.
  5. Survey 2: Detailed language background and switching comfort.
Figure 1. Overview of the study procedure. Participants selected their preferred study language, provided informed consent, and completed two surveys interleaved with two chatbot conversations. Chat1 focused on university-related topics, whereas Chat2 focused on local culture and everyday practices. PAP-U denotes Papiamentu (Curaçao), and PAP-O denotes Papiamento (Aruba).

To analyze the results, the team used an LLM-as-a-judge approach. They employed GPT-5.4-mini to identify the dominant language of every message and classify the "triggers" that caused a user to switch languages.

Datasets

The study did not use a pre-existing corpus for training but instead generated a novel dataset of interaction logs:

  • Primary Sample: 42 local students (predominantly quadrilingual).
  • Comparison Sample: 20 international students (mostly English-dominant).
  • Models Used: Participants interacted with either GPT-5.5 or Llama 4 Maverick. The system prompts explicitly instructed the models to detect the user's language and respond in kind to ensure the AI didn't "force" a specific language choice after the initial English greeting.

Key Insights and Ideas

  • Situational vs. Metaphorical Switching: Users engaged primarily in situational switching. They didn't just switch randomly; they moved toward Papiamentu/o and Spanish when the topic shifted to local culture, mirroring real-world social norms where Dutch/English are for "work" and Papiamentu/o is for "home."
  • The "English Default" Bias: Despite being quadrilingual, participants used English for the majority of messages. This suggests a "technological unmarked code"—an assumption that AI "works better" in English, likely learned from previous experiences with less capable systems.
  • Turn-by-Turn over Intra-sentential: Unlike spoken conversation where speakers might mix languages mid-sentence, these users almost exclusively switched between turns. This reflects a common pattern in digital text-based communication where the "message" is the unit of thought.
  • The Familiarity Paradox: Interestingly, participants who reported higher familiarity with AI switched languages less often. This suggests that "power users" have learned to accommodate the machine's perceived linguistic preferences to avoid errors.

Key Results

  • Topic-Driven Shifts: In the GPT group, the use of Papiamentu/o and Spanish jumped from a mere 2.8% in the university chat to 22.4% in the cultural chat.
  • English Dominance: English accounted for roughly 50% to 80% of all user messages across both topics, even though it is rarely the primary language spoken at home for these participants.
  • Switching Triggers: The vast majority of switches (89–97%) occurred when the AI asked a new question, indicating that users see a new question as an "opening" to reset the linguistic context.
  • Comfort Levels: There was no statistically significant difference in reported comfort when switching languages with an AI versus a human. Users are ready for multilingual AI, even if the AI isn't always ready for them.
Distribution of user-message languages across university-related (Chat1) and culture-related (Chat2) conversations.

Limitations and Future Work

The authors acknowledge several constraints:

  • The "English Opening": The chatbot always started in English. This likely anchored the conversation and may have suppressed more frequent switching.
  • Sample Size and Diversity: The sample was relatively small and concentrated in specific academic fields.
  • Model Confounding: Participants were not randomly assigned to GPT or Llama models, making it difficult to draw causal conclusions about model performance.
  • Future Directions: The researchers suggest that future AI evaluations must move beyond testing languages in isolation. Systems need to be tested on their ability to follow a user across language boundaries in a single session, especially when moving into low-resource languages.

Summary updated

Abstract

Multilingual speakers choose languages by topic and conversation partner. Large language models (LLMs) support these languages unequally. Research on code-switching with conversational AI has mostly examined bilingual pairs of well-resourced languages. In Curaçao and Aruba, many people use Dutch, English, Spanish, and the low-resource Creole Papiamentu/o. In a preregistered study, 42 university students held two five-minute chatbot conversations, one about university life and one about local food and culture. Surveys assessed language background, AI use, and switching comfort. 20 international students in Curaçao formed a comparison sample. Participants switched between turns and rarely mixed languages within a message. English dominated both conversations. Participants switched more often in the cultural conversation, where Papiamentu/o and Spanish rose from 15% to 24% of messages. Self-reports showed no lower comfort switching with the chatbot than with a person. Conversational agents must follow turn-by-turn language changes and handle local low-resource languages well.
BibTeX
@misc{bush2026beyond,
  author = {Annika Bush and Melika Minnaar and Salys Sultan and Tasnimul Hasan Tashin and Ankur Bhatt and Meltem Aksoy},
  title = {{Beyond Bilingual: Studying Quadrilingual Code-Switching with Conversational AI}},
  year = {2026},
  eprint = {2611.21394},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21394},
}
Figure