Spoken Language Identification Using ConvNets

Sarthak, Shikhar Shukla, Govind Mittal

TL;DR

This paper explores end-to-end deep learning approaches for Spoken Language Identification (LI), moving away from manual feature engineering. By utilizing 1D-ConvNets on raw audio and 2D-ConvNets with attention mechanisms on spectrograms, the authors achieved up to 95.4% accuracy across six languages, demonstrating that raw waveforms can bypass traditional pre-processing bottlenecks without sacrificing significant performance.

What Problem Does This Paper Address?

Language Identification is the critical first step for voice assistants (like Siri or Alexa) and automated call routing. Traditionally, this relied on extracting "prosodic" and "acoustic" features—specific units of sound called phonemes. However, many languages share overlapping phonemes (e.g., "cat" in English vs. "kat" in Dutch), making manual classification difficult and error-prone.

Furthermore, most modern systems convert audio into visual representations (spectrograms) before processing. This conversion creates a computational bottleneck that scales linearly with audio length. The authors seek to determine if neural networks can identify languages more efficiently using raw audio or enhanced visual models that "pay attention" to the most relevant parts of a signal.

Methods

The authors propose and benchmark three primary architectures:

  1. 1D-ConvNet: This model takes raw audio waveforms as input. It uses 1D convolutional layers to "auto-extract" features directly from the sound wave, bypassing the need to generate spectrograms entirely.
  2. 2D-ConvNet with Attention & Bi-directional GRU: This is the most complex model. It converts audio into log-Mel spectrograms and applies:
  3. Spatial and Channel Attention: These layers help the model focus on specific regions of the spectrogram and find dependencies between different frequency channels.
  4. Bi-directional GRU: A recurrent layer that captures temporal patterns (how the speech changes over time).
  5. Residual Connections: "Skip connections" that help train deeper networks by preventing the vanishing gradient problem.
  6. Standard 2D-ConvNet: A simplified version of the above without the attention or recurrent layers.

To improve generalization, they employed Mixup augmentation, a technique where the model is trained on linear combinations of two random images and their labels, effectively forcing the network to learn more robust boundaries between languages.

Datasets

The study utilized the VoxForge dataset, an open-source speech corpus.

  • Composition: 1,500 samples for each of the six languages: English, French, German, Spanish, Russian, and Italian.
  • Nature of Data: Because samples are submitted by users using their own microphones, the dataset contains significant variations in speech quality, making it a realistic representation of real-world scenarios.
  • Preprocessing: Audio was sampled at 8 kHz and normalized to 10-second clips (shorter clips were looped; longer clips were truncated).

Key Insights and Ideas

  • Raw Audio is Viable: The 1D-ConvNet achieved 93.7% accuracy. While slightly lower than the 2D models, it eliminates the "bottleneck" of spectrogram generation, making it a strong candidate for high-speed, real-time applications.
  • The "Phyla" Factor: Misclassifications weren't random. The models primarily struggled with languages within the same linguistic family (e.g., confusing Spanish, French, and Italian—all Romance languages). This suggests the models are successfully picking up on deep structural similarities in speech.
  • Attention Matters: Adding spatial and channel-wise attention allowed the 2D model to ignore noise and focus on the most "language-defining" parts of the spectrogram.
  • Mixup for Regularization: The use of Mixup augmentation was crucial for the 2D-ConvNet, helping it reach its peak performance by preventing overfitting on the relatively small dataset.

Key Results

The researchers benchmarked their models against several previous studies (including ResNet50 and CRNN approaches):

  • Six-Language Classification (En, Fr, De, Es, Ru, It): The 2D-ConvNet with Mixup achieved 95.4% accuracy, significantly outperforming previous benchmarks like ResNet50 (89.0%) on the same dataset.
  • Four-Language Classification (En, Fr, De, Es): The model reached 96.3% accuracy.
  • Efficiency: The 1D-ConvNet proved that end-to-end learning from raw waveforms is a competitive alternative to image-based domain conversion.

Limitations and Future Work

The authors identify several areas for improvement:

  • Silence and Noise: The current models do not explicitly remove silence from audio clips. Implementing silence removal and adding random noise during training could make the models more robust.
  • Hybrid Architectures: A promising future direction involves concatenating features from both the 1D (raw audio) and 2D (spectrogram) models to capture both high-level and low-level acoustic patterns.
  • Scaling: While six languages were tested, global systems require support for hundreds. Future work will need to address how these models scale as the number of output classes increases.

Summary updated

Abstract

Language Identification (LI) is an important first step in several speech processing systems. With a growing number of voice-based assistants, speech LI has emerged as a widely researched field. To approach the problem of identifying languages, we can either adopt an implicit approach where only the speech for a language is present or an explicit one where text is available with its corresponding transcript. This paper focuses on an implicit approach due to the absence of transcriptive data. This paper benchmarks existing models and proposes a new attention based model for language identification which uses log-Mel spectrogram images as input. We also present the effectiveness of raw waveforms as features to neural network models for LI tasks. For training and evaluation of models, we classified six languages (English, French, German, Spanish, Russian and Italian) with an accuracy of 95.4% and four languages (English, French, German, Spanish) with an accuracy of 96.3% obtained from the VoxForge dataset. This approach can further be scaled to incorporate more languages.
BibTeX
@inproceedings{sarthak2019spoken,
  author = {Sarthak and Shikhar Shukla and Govind Mittal},
  title = {{Spoken Language Identification Using ConvNets}},
  booktitle = {AmI},
  year = {2019},
  url = {https://openreview.net/forum?id=tUpM1boPTJd},
}

Loading related content...

Figure