Hierarchical multilayer perceptron based language identification

David Imseng, Mathew Magimai Doss, Hervé BourlardInterspeech 2010

TL;DR

This paper introduces a novel hierarchical multilayer perceptron (MLP) approach for language identification (LID). By processing phoneme posterior probabilities over longer temporal contexts, their method significantly outperforms traditional approaches relying on phonotactic constraints or full speech recognition. This suggests that capturing phonetic-temporal patterns is key to more robust LID.

What Problem Does This Paper Address?

Automatic Language Identification (LID) aims to determine the language of a spoken utterance. Existing LID systems often leverage various linguistic cues, such as phonological patterns (how sounds combine), morphological structures (word formation), syntax (sentence structure), and prosody (rhythm and intonation). A common and effective approach involves using phoneme recognizers, which identify basic sound units, and then applying phonotactic constraints – rules about valid sound sequences within a language. Another method uses full speech recognition systems to transcribe the speech and then identify the language. However, these methods have limitations. Phonotactic constraints are often based on limited lexical information, and speech recognition systems can be computationally expensive and prone to errors that propagate to the LID task. This paper seeks to improve LID performance by exploring a more data-driven approach that can implicitly learn complex phonetic-temporal patterns.

Methods

The core of this paper is a novel hierarchical MLP-based approach for LID. It builds upon a previously proposed hierarchical MLP for phoneme posterior estimation.

  1. Universal Phoneme Set MLP (phoneMLP): The first layer is a standard MLP trained to classify a "universal" set of phonemes (92 phonemes in this case) using acoustic features (PLP cepstral coefficients with derivatives). This layer acts as a multilingual phoneme recognizer, producing phoneme posterior probabilities. The use of a universal phoneme set allows for data sharing and discriminant training across languages, potentially aiding in bootstrapping systems for new languages.
  2. Hierarchical LID MLP (LID-MLP): The output of the phoneMLP (sequences of phoneme posterior probabilities) is then fed as input to a second MLP, the LID-MLP. This second MLP is trained to directly classify the language. Crucially, it takes a larger temporal context (varying from 10ms to 310ms) of these phoneme posteriors as input. The hypothesis is that this second MLP can learn implicit phonetic-temporal patterns, including acoustic confusions between phonemes and language-specific phonotactic information, which are indicative of a particular language.

The authors compare this hierarchical approach (System Hier) against two baseline systems:

  • System PC (Phonotactic Constraints): Uses language-specific HMM/MLP phoneme recognizers where HMM emission likelihoods are derived from the phoneMLP output. Phonotactic constraints are modeled using phoneme bigrams derived from language-specific lexicons.
  • System SR (Speech Recognition): Employs parallel language-specific hybrid HMM/MLP speech recognizers. Again, HMM emission likelihoods come from the phoneMLP. Language is determined by the word hypothesis with the highest likelihood.

Datasets

The experiments were conducted using the SpeechDat(II) corpus. Specifically, data from five European languages were used:

  • British English (EN)
  • Spanish (ES)
  • Italian (IT)
  • Swiss French (SF)
  • Swiss German (SZ)

The corpus contains read application words. For each language, a subset of 2000 speakers was used, split into training, development, and test sets to ensure comparability across languages. The authors mention using scripts to ensure reproducible splits, rather than pre-defined test sets.

Key Insights and Ideas

  • Phonetic-Temporal Patterns are Key: The most significant insight is that modeling longer temporal sequences of phoneme posterior probabilities (rather than just local acoustic features or explicit phonotactic rules) allows an MLP to learn rich, language-discriminative patterns.
  • Implicit Learning Outperforms Explicit Rules: The hierarchical MLP approach, which learns these patterns in a data-driven manner, significantly outperforms systems that explicitly encode phonotactic constraints or rely on full speech recognition. This suggests that explicit modeling might miss subtle but important discriminative information.
  • Universal Phoneme Sets are Beneficial: Using a universal phoneme set for the initial layer enables data sharing and potentially simplifies the learning process for the subsequent layers, especially for languages with fewer resources.
  • Temporal Context Matters: The performance of the hierarchical LID-MLP is highly dependent on the temporal context. There's a clear trend where increasing context improves accuracy, saturating around 230ms. This highlights the importance of capturing sequential information beyond single frames.
  • Language-Specific Phoneme Sets vs. Universal: While not directly tested as a comparison, the choice of a universal phoneme set implies a trade-off. It might lead to less precise phoneme identification for individual languages compared to language-specific recognizers, but the subsequent hierarchical layer compensates by learning language-specific patterns from these "noisy" but shared posteriors.

Key Results

  • The proposed Hierarchical MLP-based approach (System Hier) achieved the highest language identification accuracy.
  • With a temporal context of 290ms for the LID-MLP, System Hier achieved 96.4% accuracy.
  • This significantly outperformed System SR (Speech Recognition) at 94.8% and System PC (Phonotactic Constraints) at 82.3%.
  • The performance of System Hier improved with increasing temporal context, saturating around 230ms.
  • System Hier began to outperform System SR at a temporal context of around 130ms.
  • Even with zero temporal context (effectively just using current frame posteriors), the hierarchical MLP approach showed better performance than System PC.
  • Confusion analysis revealed that languages with more mono-phonemes (Italian, Swiss German) tended to have lower misclassification rates but more false positives in System Hier, possibly due to the nature of their phoneme sets.

Limitations and Future Work

  • Limited Languages: The study was conducted on only five European languages.
  • Read Speech: The experiments used read application words, not continuous spontaneous speech, which is a more challenging scenario.
  • Future Work: The authors plan to:
  • Investigate the approach with a larger number of languages.
  • Evaluate performance on continuous speech data.
  • Explore other techniques for creating universal phoneme sets.
  • Further analyze the learned phonetic-temporal patterns.

Summary updated

BibTeX
@inproceedings{imseng2010hierarchical,
  author = {David Imseng and Mathew Magimai Doss and Hervé Bourlard},
  title = {{Hierarchical multilayer perceptron based language identification}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure