Improving End-to-End Bangla Speech Recognition with Semi-supervised Training

Nafis Sadeq, Nafis Tahmid Chowdhury, Farhan Tanvir Utshaw, Shafayat Ahmed, Muhammad Abdullah AdnanEMNLP 2020 · DOI

Introduction

Building high-quality Automatic Speech Recognition (ASR) systems typically requires thousands of hours of paired audio-transcript data. For languages like Bangla, such resources are scarce; the widely used Google Bangla corpus contains only about 229 hours. However, unpaired data—audio without transcripts and text without audio—is abundant. This research addresses the challenge of leveraging these separate data sources to improve ASR performance in low-resource settings by introducing a novel semi-supervised training method.

Methodology

The proposed system utilizes a Hybrid CTC-Attention end-to-end architecture modified for semi-supervised learning. The core idea is to map both audio and text into a shared representation space.

Shared Encoding Architecture

To align audio and text representations, the model employs a shared encoder. Audio features are processed through a Pyramid BLSTM for downsampling, while text characters are converted into continuous vectors via an Embedding layer. Both streams are then fed into the shared encoder to learn a common representation space, which is then passed to a decoder for text output.

The semi-supervised training architecture featuring shared components and inter-domain loss.

Global Encoding Distance (GED) Loss

A key contribution of this work is the Global Encoding Distance (GED) loss. Traditional semi-supervised methods often use local alignment losses (like MMD or KL-divergence) that are limited by the small sample size of training minibatches, leading to unstable training.

GED overcomes this by using a pre-calculated "representative matrix" ($X$) that captures the global distribution of the entire unpaired dataset. During training, the loss measures the distance between a specific data point and its nearest neighbor in this global matrix. This ensures the model aligns data based on the global variance of the language, providing a stable "north star" for alignment regardless of batch size.

Visualization of the Global Encoding Distance (GED) loss mapping audio and text into a shared space using global context.

Experimental Setup

The model was trained using a combination of paired and unpaired datasets:

  • Paired Data: 229 hours of annotated Bangla speech from the Google Bangla corpus.
  • Unpaired Audio: 1,000 hours of news recordings crawled from Bangladeshi TV channels.
  • Unpaired Text: 800,000 Bangla sentences crawled from 40 different news websites.

Evaluation was conducted on a test set of 2,000 utterances covering various domains and speakers.

Results and Discussion

The semi-supervised approach with GED loss significantly outperformed supervised baselines and other popular inter-domain loss functions. By training a text-to-text auto-encoder alongside the speech-to-text task, the shared encoder learns the internal structure of the Bangla language, acting as an implicit language model.

Method Word Error Rate (WER)
Supervised Baseline 37.0%
Baseline + Character-level LM 32.5%
Proposed Semi-Supervised (GED Loss) 31.9%
Semi-Supervised (MMD Loss) 32.7%
Semi-Supervised (Gaussian KL-divergence) 34.0%

The GED loss proved much more robust to changes in batch size compared to MMD, making it easier to train on standard hardware. Notably, the system achieved these results without needing an external language model during final inference.

Conclusion

This research demonstrates that leveraging large-scale unpaired data through a shared encoding framework and the novel GED loss can substantially improve ASR performance for low-resource languages like Bangla. While the method requires a preprocessing step to calculate the global representative matrix, the gains in accuracy and training stability are significant. Future work will explore the integration of formal language models during the decoding phase and the application of this technique to informal or dialectal Bangla.

Summary updated

BibTeX
@inproceedings{sadeq2020improving,
  author = {Nafis Sadeq and Nafis Tahmid Chowdhury and Farhan Tanvir Utshaw and Shafayat Ahmed and Muhammad Abdullah Adnan},
  title = {{Improving End-to-End Bangla Speech Recognition with Semi-supervised Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.169},
}
Figure