Nepali-English Code-Mixed Speech Recognition: Data Preparation and Adaptation of a Speech Foundation Model

Jeevan Neupane, Janardan Bhetwal, Kshitiz Paudel, Anup Aryal, Bipin BashyalarXiv 2026 · arXiv

This paper presents a robust pipeline for adapting the SraVaani foundation model to Nepali–English code-mixed speech—a common linguistic phenomenon where speakers switch between languages within a single sentence. By introducing a novel audio-guided method to handle numbers and symbols within a fixed tokenizer, the authors achieved a massive reduction in Word Error Rate (WER), reaching performance parity with models nearly twice its size, like Whisper large-v3-turbo.

TL;DR

The authors successfully adapted the 449M-parameter SraVaani model for Nepali–English code-mixed ASR, reducing WER from 48.6% to 16.3%. Their key innovation is an audio-guided "bilingual number verbalization" technique that allows the model to handle digits and symbols despite a fixed tokenizer that lacks them.

What Problem Does This Paper Address?

Nepali speakers frequently mix English words into their daily conversations, especially in technical and educational contexts. This "code-mixing" presents two major challenges for Automatic Speech Recognition (ASR):

  1. Script Switching: Transcripts are expected to use Devanagari for Nepali and Latin script for English. Foundation models often struggle to switch scripts correctly or default to transliterating English into Devanagari.
  2. Tokenizer Limitations: Many high-performance models use fixed tokenizers that do not support digits or common symbols (like % or +). Standard practice is to discard such data or use simple rules to spell out numbers. However, in code-mixed speech, a number like "1947" could be read in Nepali or English, and a simple rule-based approach often leads to mismatches between the text labels and the actual audio.

Methods

The researchers adapted SraVaani, a hybrid model combining a FastConformer encoder with Token-and-Duration Transducer (TDT) and CTC output heads.

Audio-Guided Bilingual Number Verbalization

To solve the "number problem" without rebuilding the tokenizer, the authors developed a three-step pipeline:

  • Candidate Generation: For every digit/symbol in the source transcript, they generated all possible Nepali and English verbalizations (e.g., "10%" becomes "दस प्रतिशत", "दस percent", or "ten percent").
  • Phonetic Evidence: They ran the base SraVaani model on the audio. Even if the model got the script wrong, its phonetic output provided a "hint" of what was actually said.
  • Selection: They romanized both the candidates and the model's output, then used Levenshtein distance to pick the candidate that best matched the audio.
Figure 2: Spelling out numbers, on a real lecture clip. Rules propose a Nepali and an English reading for each number. We romanise the candi- dates and the base model’s output, which writes English in Devanagari, and keep the closest candidate.

Full Fine-Tuning

Unlike many adaptation studies that freeze the encoder to save compute, the authors performed full fine-tuning of all 449M parameters. They used a cosine learning rate schedule and found that updating the encoder was critical for the model to learn the acoustic transitions between languages.

Figure 3: The fine-tuning architecture for SraVaani, utilizing both TDT and CTC losses. (Note: Figure 3 in the paper corresponds to the fine-tuning step)

Figure 2: Spelling out numbers, on a real lecture clip. Rules propose a Nepali and an English reading for each number. We romanise the candi- dates and the base model’s output, which writes English in Devanagari, and keep the closest candidate.

Datasets

The study utilized a 172.8-hour corpus covering six domains:

  • OpenSLR Nepali: Crowd-sourced read speech (mostly monolingual).
  • YouTube-sourced data: Engineering lectures (highly technical/English-heavy), podcasts, news/talk shows, spontaneous conversations, and technical interviews.

For evaluation, they used an in-domain test set and the public FLEURS Nepali dataset. Crucially, they created a "mixed-script" version of FLEURS to better evaluate code-mixed performance.

Key Insights and Ideas

  • Encoder Adaptation is Essential: Freezing the encoder resulted in a WER of 25.5%, compared to 17.1% with full adaptation. This suggests the encoder needs to learn the specific acoustic "flavor" of code-switching, not just the output script.
  • Numeric Data is a Performance Multiplier: Including numeric clips in training (enabled by their verbalization technique) reduced WER on numeric test speech from 21.3% to 14.3%.
  • The "Convention" Trap: Standard benchmarks like FLEURS often write English loanwords in Devanagari. Evaluating a model that correctly outputs Latin script against these references artificially inflates error rates. Using their "mixed-script" reference set reversed the apparent performance loss on out-of-domain data.
  • Monolingual Data Still Matters: Removing the monolingual OpenSLR data from training increased WER on the code-mixed domains, showing that high-quality read speech helps preserve general recognition capabilities.

Key Results

The fine-tuned SraVaani model showed dramatic improvements across the board:

  • Overall WER: Dropped from 48.6% (zero-shot) to 16.3% after 12 epochs of fine-tuning.
  • English Word Recognition: English-word WER fell from 80.4% to 18.8%.
  • Comparison: SraVaani (449M params) achieved 16.3% WER, effectively matching Whisper large-v3-turbo (809M params) at 16.4% on the same test set.
Figure 4: Six-epoch validation WER for the main configuration and ablations. Validation is performed twice per epoch.

Limitations and Future Work

  • Speaker Overlap: The train/test splits are not guaranteed to be speaker-disjoint, which may lead to slightly optimistic in-domain results.
  • Label Bias: Selecting number readings based on the base model's output might reinforce the model's existing errors.
  • Spontaneous Out-of-Domain Data: FLEURS consists of read speech; the model's performance on truly spontaneous, out-of-domain code-mixed speech remains an open question.
  • Future Directions: The authors suggest verifying numeric labels independently and training with multiple accepted transcription conventions to improve robustness.

Summary updated

Abstract

Nepali speakers regularly mix English words into conversation, but multilingual speech recognisers often struggle with language switches and the writing script used for English words. We study how to adapt SraVaani, a 449M-parameter hybrid TDT-CTC model, using a 172.8-hour corpus covering six Nepali speech domains. Its pretrained tokenizer does not support digits and several common symbols. Rather than replace it or discard these recordings, we generate Nepali and English readings for numbers and use zero-shot ASR output to select a plausible form. Full fine-tuning reduces test word error rate (WER) from 48.6% to 17.1%; a 12-epoch run reaches 16.3%, compared with 16.4% for fine-tuned Whisper large-v3-turbo. Freezing the encoder increases WER to 25.5%, while excluding numeric clips raises WER on numeric test speech from 14.3% to 21.3%. We also evaluate on FLEURS using both its original references and mixed-script alternatives. The comparison shows that transcription conventions explain part, but not all, of the out-of-domain performance loss.
BibTeX
@misc{neupane2026nepali,
  author = {Jeevan Neupane and Janardan Bhetwal and Kshitiz Paudel and Anup Aryal and Bipin Bashyal},
  title = {{Nepali-English Code-Mixed Speech Recognition: Data Preparation and Adaptation of a Speech Foundation Model}},
  year = {2026},
  eprint = {2611.13512},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13512},
}
Figure