This paper presents a robust pipeline for adapting the SraVaani foundation model to Nepali–English code-mixed speech—a common linguistic phenomenon where speakers switch between languages within a single sentence. By introducing a novel audio-guided method to handle numbers and symbols within a fixed tokenizer, the authors achieved a massive reduction in Word Error Rate (WER), reaching performance parity with models nearly twice its size, like Whisper large-v3-turbo.
TL;DR
The authors successfully adapted the 449M-parameter SraVaani model for Nepali–English code-mixed ASR, reducing WER from 48.6% to 16.3%. Their key innovation is an audio-guided "bilingual number verbalization" technique that allows the model to handle digits and symbols despite a fixed tokenizer that lacks them.
What Problem Does This Paper Address?
Nepali speakers frequently mix English words into their daily conversations, especially in technical and educational contexts. This "code-mixing" presents two major challenges for Automatic Speech Recognition (ASR):
- Script Switching: Transcripts are expected to use Devanagari for Nepali and Latin script for English. Foundation models often struggle to switch scripts correctly or default to transliterating English into Devanagari.
- Tokenizer Limitations: Many high-performance models use fixed tokenizers that do not support digits or common symbols (like
%or+). Standard practice is to discard such data or use simple rules to spell out numbers. However, in code-mixed speech, a number like "1947" could be read in Nepali or English, and a simple rule-based approach often leads to mismatches between the text labels and the actual audio.
Methods
The researchers adapted SraVaani, a hybrid model combining a FastConformer encoder with Token-and-Duration Transducer (TDT) and CTC output heads.
Audio-Guided Bilingual Number Verbalization
To solve the "number problem" without rebuilding the tokenizer, the authors developed a three-step pipeline:
- Candidate Generation: For every digit/symbol in the source transcript, they generated all possible Nepali and English verbalizations (e.g., "10%" becomes "दस प्रतिशत", "दस percent", or "ten percent").
- Phonetic Evidence: They ran the base SraVaani model on the audio. Even if the model got the script wrong, its phonetic output provided a "hint" of what was actually said.
- Selection: They romanized both the candidates and the model's output, then used Levenshtein distance to pick the candidate that best matched the audio.
Full Fine-Tuning
Unlike many adaptation studies that freeze the encoder to save compute, the authors performed full fine-tuning of all 449M parameters. They used a cosine learning rate schedule and found that updating the encoder was critical for the model to learn the acoustic transitions between languages.
Figure 3: The fine-tuning architecture for SraVaani, utilizing both TDT and CTC losses. (Note: Figure 3 in the paper corresponds to the fine-tuning step)
Datasets
The study utilized a 172.8-hour corpus covering six domains:
- OpenSLR Nepali: Crowd-sourced read speech (mostly monolingual).
- YouTube-sourced data: Engineering lectures (highly technical/English-heavy), podcasts, news/talk shows, spontaneous conversations, and technical interviews.
For evaluation, they used an in-domain test set and the public FLEURS Nepali dataset. Crucially, they created a "mixed-script" version of FLEURS to better evaluate code-mixed performance.
Key Insights and Ideas
- Encoder Adaptation is Essential: Freezing the encoder resulted in a WER of 25.5%, compared to 17.1% with full adaptation. This suggests the encoder needs to learn the specific acoustic "flavor" of code-switching, not just the output script.
- Numeric Data is a Performance Multiplier: Including numeric clips in training (enabled by their verbalization technique) reduced WER on numeric test speech from 21.3% to 14.3%.
- The "Convention" Trap: Standard benchmarks like FLEURS often write English loanwords in Devanagari. Evaluating a model that correctly outputs Latin script against these references artificially inflates error rates. Using their "mixed-script" reference set reversed the apparent performance loss on out-of-domain data.
- Monolingual Data Still Matters: Removing the monolingual OpenSLR data from training increased WER on the code-mixed domains, showing that high-quality read speech helps preserve general recognition capabilities.
Key Results
The fine-tuned SraVaani model showed dramatic improvements across the board:
- Overall WER: Dropped from 48.6% (zero-shot) to 16.3% after 12 epochs of fine-tuning.
- English Word Recognition: English-word WER fell from 80.4% to 18.8%.
- Comparison: SraVaani (449M params) achieved 16.3% WER, effectively matching Whisper large-v3-turbo (809M params) at 16.4% on the same test set.
Limitations and Future Work
- Speaker Overlap: The train/test splits are not guaranteed to be speaker-disjoint, which may lead to slightly optimistic in-domain results.
- Label Bias: Selecting number readings based on the base model's output might reinforce the model's existing errors.
- Spontaneous Out-of-Domain Data: FLEURS consists of read speech; the model's performance on truly spontaneous, out-of-domain code-mixed speech remains an open question.
- Future Directions: The authors suggest verifying numeric labels independently and training with multiple accepted transcription conventions to improve robustness.
Summary updated