When humans fluidly mix multiple languages within a single conversation—a phenomenon known as code-switching—they do not adhere to rigid dictionary boundaries. Instead, they blend scripts, adapt phonetics, and adopt loanwords dynamically.
Traditional evaluation metrics for language and speech systems, such as Word Error Rate (WER) for Automatic Speech Recognition (ASR) or BLEU for text generation, rely heavily on exact string matching against a single reference text. Because of this, they fail dramatically when applied to code-switched data. The core of this research direction focuses on diagnosing why traditional metrics break down in multilingual contexts and developing evaluation frameworks that align with actual human comprehension.
Why Traditional Metrics Fail
Standard metrics operate on the assumption that there is only one correct orthographic representation for a spoken word. In code-switching, this assumption falls apart due to three main factors: script variation, loanword integration, and morphological blending.
For example, if an Arabic-English speaker inserts an English word into an Arabic sentence, an ASR system might transcribe that word phonetically using Arabic script, or it might use the standard English Latin script. As Injy Hamed et al. observe in their work on dialectal Arabic-English speech, standard WER penalizes these valid phonetic transcriptions as outright errors if they do not match the arbitrary script choice of the human annotator.
This creates an artificial inflation of error rates. In a comprehensive survey of code-switched Arabic NLP, Injy Hamed et al. emphasize that metrics like WER and BLEU penalize the natural linguistic fluidity and acceptable orthographic diversity of informal, multi-script communication. The metrics measure surface-level string deviations rather than communicative competence.
Transliteration-Aware Evaluation
To resolve the script mismatch problem, researchers have designed metrics that normalize text into a shared representation before calculating error rates. Instead of comparing raw strings, these metrics evaluate the underlying phonetic or semantic intent.
[Spoken Code-Switched Utterance]
│
▼
┌──────────────────────────────┐
│ ASR Hypothesis Token │
│ (e.g., English in Arabic) │
└──────────────┬───────────────┘
│
▼ [Transliteration / Script Mapping]
┌──────────────────────────────┐
│ Normalized Target Script │ ◄─── Compare using T-WER / TW
└──────────────────────────────┘
▲
│ [Transliteration / Script Mapping]
┌──────────────┴───────────────┐
│ Reference Token │
│ (e.g., English in Latin) │
└──────────────────────────────┘
In the context of low-resource Indian languages, where speakers frequently alternate between regional scripts and Latin text, Anuj Diwan et al. introduced Transliterated Word Error Rate (T-WER). This metric accommodates variations in script usage by allowing words to be transliterated from one script to another within the same utterance before error calculation, preventing models from being penalized for valid cross-script rendering.
Similarly, Shammur Absar Chowdhury et al. developed a transliterated WER (TW) metric for Arabic ASR. Their framework converts recognized foreign language tokens into the target Arabic script. By mitigating rendering inconsistencies, the authors demonstrate a more nuanced and accurate assessment of model capabilities. Ahmed Ali et al. also point to transliterated word error rates as a vital tool for capturing the intricate nuances of intrasentential code-switching, where language switches occur rapidly within a single sentence.
Aligning with Human Acceptability
While transliteration fixes surface-level script mismatches, it does not capture whether a code-switched sentence sounds natural or acceptable to a native speaker. Code-switching is governed by complex internal grammatical intuitions, not just random mixing.
Historically, researchers used structural metrics like the Code Mixing Index (CMI), Burstiness, or the number of switch points to quantify code-switching. However, Prashant Kodali et al. demonstrated that these traditional metrics correlate very poorly with actual human judgments of sentence acceptability. To fix this, the authors introduced a large-scale dataset of human acceptability judgments, showing that fine-tuned multilingual language models predict human linguistic intuition far better than any structural formula.
To ground speech evaluation in human reality, Injy Hamed et al. introduced the Human Acceptability Corpus (HAC), a benchmark containing human-edited ASR hypotheses. By evaluating how humans rate imperfect transcriptions, the authors proved that metrics incorporating a combination of transliteration, phonological similarity, and semantic translation correlate far better with human utility judgments than standard string-matching algorithms.
The Downstream Impact of Poor Metrics
The inadequacy of baseline evaluation metrics creates a cascading problem for downstream speech applications. If an evaluation framework cannot reliably measure transcription quality, it cannot accurately measure the performance of systems built on top of that transcription.
This bottleneck is explicitly highlighted by Sarina Meyer et al. in their investigation of voice anonymization for code-switched speech. To evaluate whether anonymized speech remains useful, researchers typically pass the altered audio through an ASR model and measure the output quality. However, because standard ASR models inherently exhibit high error rates on spontaneous, mixed-language data, the resulting utility scores become skewed and unreliable. Developing robust, code-switching-compliant evaluation frameworks is therefore not just an academic exercise; it is a prerequisite for deploying reliable, private, and inclusive speech technologies in multilingual societies.
Key Foundational Work
Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR (2021) — contributed 4 concept elements to this direction.
Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition (2022) — contributed 3 concept elements to this direction.
MUCS 2021: Multilingual and Code-Switching ASR Challenges for Low Resource Indian Languages (2021) — contributed 3 concept elements to this direction.