Spoken Language Identification using ConvNets

Sarthak, Shikhar Shukla, Govind Mittal

Abstract

Language Identification (LI) is an important first step in several speech processing systems. With a growing number of voice-based assistants, speech LI has emerged as a widely researched field. To approach the problem of identifying languages, we can either adopt an implicit approach where only the speech for a language is present or an explicit one where text is available with its corresponding transcript. This paper focuses on an implicit approach due to the absence of transcriptive data. This paper benchmarks existing models and proposes a new attention based model for language identification which uses log-Mel spectrogram images as input. We also present the effectiveness of raw waveforms as features to neural network models for LI tasks. For training and evaluation of models, we classified six languages (English, French, German, Spanish, Russian and Italian) with an accuracy of 95.4% and four languages (English, French, German, Spanish) with an accuracy of 96.3% obtained from the VoxForge dataset. This approach can further be scaled to incorporate more languages.

Introduction

Language Identification (LI) serves as a foundational component in modern speech processing pipelines, particularly for voice-activated assistants and automated customer service systems. Traditionally, LI relied on manual feature extraction of prosodic and acoustic units like phonemes. However, this approach is limited by the overlap of phonemes across different languages, leading to classification challenges. For instance, similar sounds in English and Dutch can lead to misclassification if the system relies solely on predefined phonetic units. This paper explores an implicit approach to LI, leveraging deep learning to automatically extract features from raw audio and spectrograms, thereby bypassing the limitations and computational bottlenecks of manual engineering.

Methodology

The research benchmarks three primary architectures: a 1D-ConvNet for raw audio processing, and two 2D-ConvNet variants for log-Mel spectrogram analysis. The 1D-ConvNet architecture utilizes a series of one-dimensional convolutional layers to learn hierarchical representations directly from the time-domain waveform. This end-to-end approach is particularly advantageous for real-time applications where the latency of generating spectrograms is undesirable.

In contrast, the 2D-ConvNet models leverage the visual patterns in log-Mel spectrograms. The most advanced variant incorporates spatial and channel-wise attention mechanisms, allowing the network to focus on the most "language-defining" parts of the signal while ignoring background noise. It also utilizes a Bi-directional GRU layer to capture temporal patterns in speech and residual connections to facilitate the training of deeper networks.

To optimize these models, extensive hyperparameter tuning was conducted across various configurations. Figure 1 illustrates the impact of parameters such as the number of filters, kernel size, dropout rates, and batch size on validation accuracy. These experiments were crucial in determining the final architecture that balances model complexity with generalization performance.

Hyperparameter optimization results showing validation accuracy across different model configurations.

The models were trained on the VoxForge dataset, an open-source corpus containing speech samples with significant real-world variations in recording quality. The study focused on six languages: English, French, German, Spanish, Russian, and Italian. Audio was normalized to 10-second clips at 8 kHz. To further enhance robustness, Mixup augmentation was employed, training the models on linear combinations of samples to create smoother decision boundaries.

Results and Comparative Analysis

The proposed models demonstrated state-of-the-art performance, significantly outperforming previous benchmarks. As shown in the comparative analysis in Figure 4, the 2D-ConvNet with Mixup achieved a peak accuracy of 95.4% for six-language classification, a notable improvement over the 89.0% achieved by ResNet50 on the same dataset. For a four-language subset, the model reached 96.3% accuracy.

Quantitative review of previous studies compared to the proposed ConvNet models.

The 1D-ConvNet's performance of 93.7% is particularly significant. It validates that raw waveforms can serve as highly effective features, offering a competitive alternative to image-based methods while eliminating the spectrogram generation bottleneck.

Analysis of misclassifications revealed that errors were not random but often occurred between languages within the same linguistic family, such as Romance languages (Spanish, French, and Italian). This suggests that the models are successfully picking up on deep structural similarities in speech. The inclusion of attention mechanisms was particularly effective in helping the 2D models disregard noise and focus on salient acoustic features.

Conclusion

This study confirms that deep convolutional neural networks can effectively perform Spoken Language Identification with high accuracy using both raw audio and spectral representations. The success of the 1D-ConvNet highlights a path toward more efficient, real-time LI systems. Future work will focus on improving robustness to noise and silence through advanced preprocessing and exploring hybrid architectures that combine raw audio and spectral features to further enhance classification performance across a broader range of global languages.

Summary updated

Abstract

Language Identification (LI) is an important first step in several speech processing systems. With a growing number of voice-based assistants, speech LI has emerged as a widely researched field. To approach the problem of identifying languages, we can either adopt an implicit approach where only the speech for a language is present or an explicit one where text is available with its corresponding transcript. This paper focuses on an implicit approach due to the absence of transcriptive data. This paper benchmarks existing models and proposes a new attention based model for language identification which uses log-Mel spectrogram images as input. We also present the effectiveness of raw waveforms as features to neural network models for LI tasks. For training and evaluation of models, we classified six languages (English, French, German, Spanish, Russian and Italian) with an accuracy of 95.4% and four languages (English, French, German, Spanish) with an accuracy of 96.3% obtained from the VoxForge dataset. This approach can further be scaled to incorporate more languages.
BibTeX
@inproceedings{sarthak2019spoken,
  author = {Sarthak and Shikhar Shukla and Govind Mittal},
  title = {{Spoken Language Identification using ConvNets}},
  booktitle = {AmI},
  year = {2019},
  url = {https://openreview.net/forum?id=tUpM1boPTJd},
}
Figure