Introduction
In multilingual societies, speakers frequently switch between languages within a single conversation—a phenomenon known as code-mixing. Despite its prevalence, most goal-oriented conversational AI systems are trained on monolingual data, limiting their effectiveness in real-world multilingual contexts. Researchers from IIT Madras have introduced a new benchmark dataset for goal-oriented dialogue in four Indian languages—Hindi, Bengali, Gujarati, and Tamil—code-mixed with English. By adapting the well-known DSTC2 restaurant reservation dataset, they provide a 5-way parallel corpus to facilitate the development of more robust multilingual virtual assistants.
The Challenge of Code-Mixing
Modern conversational agents like Siri or Alexa are predominantly optimized for monolingual interactions. However, in regions like India, users naturally blend their native "Matrix" language with English "Embedding" words. For instance, a user might ask: "Kya tum is restaurant mein ek table book karne mein meri help karoge?" (Will you help me book a table in this restaurant?).
Current task-oriented systems struggle with this linguistic fluidity because they lack training data that reflects how people actually communicate. This paper addresses the gap by creating a structured, task-based dataset that captures these nuances.
Methodology
The researchers employed a "template-and-instantiate" workflow to ensure both quality and scalability:
- Template Extraction: 3,590 unique sentence structures were identified from the original English DSTC2 dataset, with specific entities (like
[CUISINE]or[LOCATION]) replaced by placeholders. - Crowdsourced Translation: These templates were translated into code-mixed versions using Amazon Mechanical Turk and in-house experts. Crucially, they used Romanized script (writing Hindi or Tamil using English letters), reflecting the standard for informal digital chat.
- Naturalness Constraints: Workers were encouraged to maintain English terms where they felt natural, avoiding forced translations to capture the "vibe" of real-world code-mixing.
- Re-instantiation: The code-mixed templates were repopulated with original entities to form complete, coherent dialogues.
Dataset Characteristics
The resulting corpus, spanning Hindi, Bengali, Gujarati, and Tamil, contains approximately 49,167 utterances across 1,168 training dialogues. A key feature of this dataset is the distribution of code-mixed words per utterance, which illustrates the varying degrees of language switching across the four languages.
Key Insights
- The "Matrix" Language Challenge: Traditional measures of code-mixing often assume the language with the most words is the "base." The authors argue this is often incorrect in technical domains where English nouns may outnumber native verbs, even if the grammatical structure remains native.
- Morpheme-level Mixing: The dataset captures complex mixing, such as attaching native suffixes to English words (e.g., in Tamil, "address" becomes "addressum").
- Romanization is Essential: By using Romanized text, the dataset aligns with how users actually type on mobile devices, bypassing the complexities of script-switching.
Results and Baselines
The dataset was benchmarked using Sequence-to-Sequence (Seq2Seq) with Attention and Hierarchical Recurrent Encoder-Decoder (HRED) models.
- BLEU-4 Scores: Performance generally ranged between 53.8 and 62.1.
- Per-Dialog Accuracy: This "gold standard" for goal-oriented bots—measuring if the bot got the entire conversation right—was remarkably low, ranging from 1.0% to 1.6%.
These results highlight the significant difficulty current state-of-the-art models face with multi-turn code-mixed interactions, suggesting that this dataset should serve as a springboard for developing new architectures specifically designed for linguistic code-switching.
Limitations and Future Work
While the dataset provides a vital resource, its synthetic origin (translating an English corpus) may not fully capture unique cultural conversational styles. Additionally, the data is currently limited to the restaurant reservation domain. Future research will need to expand into other sectors like healthcare or finance and address the "accuracy gap" revealed by the baseline models.
Summary updated