Introduction
Code-switching—the linguistic phenomenon where speakers alternate between two or more languages within a single conversation—presents a significant challenge for modern Automatic Speech Recognition (ASR) systems. In the context of Gujarati-English speech, phonetic similarities between the two languages often lead to script assignment errors, even when the underlying acoustic recognition is accurate. This paper investigates methods to improve ASR performance by conditioning Transformer layers on predicted Language IDs (LID) and introduces a novel loss function to handle language alignment.
Methodology
The research utilizes a hybrid Transformer-based encoder-decoder architecture. To address the challenges of code-switching, the authors introduce several architectural innovations focused on language-specific processing.
Language-Specific Gating
A core contribution is the introduction of language-specific parameters into the multi-head attention mechanism via a gating layer. Two primary methods were explored:
- Pre-Attention Gating: This involves separate Query, Key, and Value ($Q, K, V$) embeddings for each language, which are linearly interpolated based on the output of a gating layer.
- Post-Attention Gating: Attention encodings are calculated for each language independently and then interpolated.
Seamless Temporal Classification (STC) Loss
Standard Connectionist Temporal Classification (CTC) loss often struggles with LID sequences due to the requirement for "blank" tokens to handle repetitions. In code-switched data, LID sequences are typically long and highly repetitive. The proposed Seamless Temporal Classification (STC) Loss eliminates the need for blank tokens by maximizing the likelihood of all possible input alignments that "stretch" the output sequence to match the input length.
Experimental Setup
The study was conducted using approximately 200 hours of audio data, comprising 100 hours of monolingual Gujarati and 100 hours of code-switched Gujarati-English (with Gujarati as the matrix language). Evaluation was performed on a 2-hour code-switched validation set (d-cs), an unseen 2-hour code-switched test set (t-cs), and a 2-hour monolingual dataset (d-mono).
Results and Analysis
The experiments yielded several key insights into the nature of code-switching ASR.
The Phoneme Hypothesis
The authors observed that the baseline model often recognized the correct sounds but failed to assign them to the correct language's script. To test this "Phoneme Hypothesis," a Weighted Finite State Transducer (WFST) was used to transliterate between English and Gujarati. When both the output and reference were normalized to a common phoneme set, the Word Error Rate (WER) dropped significantly, confirming that the primary bottleneck was language identification rather than acoustic modeling.
Performance and Efficiency
Despite the architectural enhancements, the proposed methods did not significantly outperform the baseline WER of 32.5% on the code-switched test set, with most variations ranging between 32.8% and 34%. However, the gating mechanism proved highly parameter-efficient, increasing the total parameter count by only 1.1% per layer. Furthermore, qualitative analysis showed that the gating weights aligned closely with actual language switches in the audio, demonstrating the model's ability to detect language transitions directly from acoustic features.
Conclusion
While the proposed methods did not yield a breakthrough in WER, they provided valuable insights into the mechanisms of language identification within Transformer-based ASR. The successful implementation of language-specific gating and the introduction of STC loss offer a foundation for future work. Future research may focus on optimizing the STC loss for better computational efficiency and exploring these architectures in multi-lingual settings beyond bilingual code-switching.
Summary updated