Gujarati-English Code-Switching Speech Recognition using ensemble prediction of spoken language

Yash Sharma, Basil Abraham, Preethi JyothiarXiv 2024 · DOI · arXiv

Introduction

Code-switching—the linguistic phenomenon where speakers alternate between two or more languages within a single conversation—presents a significant challenge for modern Automatic Speech Recognition (ASR) systems. In the context of Gujarati-English speech, phonetic similarities between the two languages often lead to script assignment errors, even when the underlying acoustic recognition is accurate. This paper investigates methods to improve ASR performance by conditioning Transformer layers on predicted Language IDs (LID) and introduces a novel loss function to handle language alignment.

Methodology

The research utilizes a hybrid Transformer-based encoder-decoder architecture. To address the challenges of code-switching, the authors introduce several architectural innovations focused on language-specific processing.

Language-Specific Gating

A core contribution is the introduction of language-specific parameters into the multi-head attention mechanism via a gating layer. Two primary methods were explored:

  1. Pre-Attention Gating: This involves separate Query, Key, and Value ($Q, K, V$) embeddings for each language, which are linearly interpolated based on the output of a gating layer.
  2. Post-Attention Gating: Attention encodings are calculated for each language independently and then interpolated.
Proposed architecture featuring the CLSR (Code-switching Language Specific Representation) layer with language-specific gating.

Seamless Temporal Classification (STC) Loss

Standard Connectionist Temporal Classification (CTC) loss often struggles with LID sequences due to the requirement for "blank" tokens to handle repetitions. In code-switched data, LID sequences are typically long and highly repetitive. The proposed Seamless Temporal Classification (STC) Loss eliminates the need for blank tokens by maximizing the likelihood of all possible input alignments that "stretch" the output sequence to match the input length.

Experimental Setup

The study was conducted using approximately 200 hours of audio data, comprising 100 hours of monolingual Gujarati and 100 hours of code-switched Gujarati-English (with Gujarati as the matrix language). Evaluation was performed on a 2-hour code-switched validation set (d-cs), an unseen 2-hour code-switched test set (t-cs), and a 2-hour monolingual dataset (d-mono).

Results and Analysis

The experiments yielded several key insights into the nature of code-switching ASR.

The Phoneme Hypothesis

The authors observed that the baseline model often recognized the correct sounds but failed to assign them to the correct language's script. To test this "Phoneme Hypothesis," a Weighted Finite State Transducer (WFST) was used to transliterate between English and Gujarati. When both the output and reference were normalized to a common phoneme set, the Word Error Rate (WER) dropped significantly, confirming that the primary bottleneck was language identification rather than acoustic modeling.

Word Error Rate (WER) performance across different normalization modes, supporting the Phoneme Hypothesis.

Performance and Efficiency

Despite the architectural enhancements, the proposed methods did not significantly outperform the baseline WER of 32.5% on the code-switched test set, with most variations ranging between 32.8% and 34%. However, the gating mechanism proved highly parameter-efficient, increasing the total parameter count by only 1.1% per layer. Furthermore, qualitative analysis showed that the gating weights aligned closely with actual language switches in the audio, demonstrating the model's ability to detect language transitions directly from acoustic features.

Conclusion

While the proposed methods did not yield a breakthrough in WER, they provided valuable insights into the mechanisms of language identification within Transformer-based ASR. The successful implementation of language-specific gating and the introduction of STC loss offer a foundation for future work. Future research may focus on optimizing the STC loss for better computational efficiency and exploring these architectures in multi-lingual settings beyond bilingual code-switching.

Summary updated

Abstract

An important and difficult task in code-switched speech recognition is to recognize the language, as lots of words in two languages can sound similar, especially in some accents. We focus on improving performance of end-to-end Automatic Speech Recognition models by conditioning transformer layers on language ID of words and character in the output in an per layer supervised manner. To this end, we propose two methods of introducing language specific parameters and explainability in the multi-head attention mechanism, and implement a Temporal Loss that helps maintain continuity in input alignment. Despite being unable to reduce WER significantly, our method shows promise in predicting the correct language from just spoken data. We introduce regularization in the language prediction by dropping LID in the sequence, which helps align long repeated output sequences.
BibTeX
@misc{sharma2024gujarati,
  author = {Yash Sharma and Basil Abraham and Preethi Jyothi},
  title = {{Gujarati-English Code-Switching Speech Recognition using ensemble prediction of spoken language}},
  year = {2024},
  eprint = {2403.08011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.08011},
}
Figure