Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNet

Manish Dhakal, Arman Chhetri, Aman Kumar Gupta, Prabin Lamichhane, Suraj Pandey, Subarna ShakyaarXiv 2024 · DOI · arXiv

Overview

Researchers have developed a high-performing end-to-end Automatic Speech Recognition (ASR) system specifically for the Nepali language. By combining 1D-Convolutional Neural Networks (CNN), Residual Networks (ResNet), and Bidirectional LSTMs, the team achieved a Character Error Rate (CER) of 17.06%, outperforming several baseline architectures.

The Challenge of Nepali ASR

While ASR technology has matured significantly for languages like English and Mandarin, "low-resource" languages like Nepali often lag behind. Previous attempts at Nepali ASR were limited by small datasets or architectures that couldn't handle the complexity of the language's phonetic structure.

The authors identify two primary hurdles:

  1. Data Quality: Raw audio datasets often contain long silent gaps at the beginning and end of clips, which confuses the model during the "alignment" phase (matching specific sounds to specific characters).
  2. Model Degradation: As deep learning models get deeper to capture more complex language patterns, they often suffer from vanishing gradients or accuracy saturation, where adding more layers actually makes the model perform worse.

Methodology

The authors proposed an end-to-end architecture that moves away from traditional phonetic modeling toward a purely deep-learning approach.

The training pipeline for the Nepali ASR system, highlighting preprocessing and model updates.

Preprocessing and Feature Extraction

  • Silence Clipping Algorithm: A custom preprocessing step uses a sliding window to calculate local means of audio intensity. If the local mean is lower than the global mean, the segment is clipped. This reduced the dataset to 44.3% of its original size, making training significantly faster and more stable.
  • Feature Extraction: The system uses 13 Mel Frequency Cepstral Coefficients (MFCCs) to convert raw audio waves into a mathematical representation that mimics human hearing.

Hybrid Architecture

The model integrates three distinct neural network components:

  • 1D-CNN: Extracts localized features from the audio signal.
  • ResNet Blocks: Five residual blocks allow the network to be deeper without losing information, effectively solving the degradation problem.
  • BiLSTM: Two Bidirectional Long Short-Term Memory layers capture the temporal context of speech, looking at both past and future "frames" to predict the current character.
Architecture of the residual block used to prevent model degradation in deep layers.

The model uses Connectionist Temporal Classification (CTC) Loss to handle the alignment between audio and text of different lengths. During inference, a Beam Search decoder finds the most likely sequence of Nepali characters.

Dataset and Training

The model was trained and tested on the OpenSLR Nepali dataset, which contains 157,905 audio clips from 527 unique speakers.

  • Cleaning: The authors removed instances containing numeric transcriptions (e.g., "1924") because they were too sparse to train effectively.
  • Final Set: After cleaning and silence clipping, they utilized 63.6 hours of audio, split into 95% for training and 5% for testing.

Key Results and Insights

The proposed 1D-CNN + ResNet + BiLSTM model emerged as the clear winner:

  • Character Error Rate (CER): 17.06% on the test set.
  • Comparison: This significantly outperformed the standalone BiLSTM (19.71% CER) and a 1D-CNN + GRU model (23.72% CER).
  • Efficiency: By clipping silent gaps, the researchers created a more "uniform mapping," allowing the CTC loss to saturate much faster.
  • The Role of ResNet: A simple 1D-CNN + BiLSTM stack actually performed worse than a standalone BiLSTM. It was only after adding Residual Blocks (providing "shortcuts" for data flow) that the deeper architecture surpassed the baseline.
  • Model Size: The final model is relatively lightweight with 1.55 million parameters, allowing it to run efficiently on standard hardware like an NVIDIA Tesla T4.

Limitations and Future Work

The current model cannot transcribe numbers, as these were removed during the data cleaning phase to maintain model stability. Additionally, while character-level prediction is robust, performance on "crowdsourced" out-of-distribution data shows some phonetic substitutions, suggesting a need for more diverse training data.

Future directions include exploring attention mechanisms (like Transformers) and expanding the dataset to include more varied accents and noisy environments.

Source Code: GitHub - ASR-Nepali-using-CNN-BiLSTM-ResNet

Summary updated

BibTeX
@misc{dhakal2024automatic,
  author = {Manish Dhakal and Arman Chhetri and Aman Kumar Gupta and Prabin Lamichhane and Suraj Pandey and Subarna Shakya},
  title = {{Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNet}},
  year = {2024},
  eprint = {2406.17825},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.17825},
}
Figure