Exploiting foreign resources for DNN-based ASR

Petr Motlícek, David Imseng, Blaise Potard, Philip N. Garner, Ivan HimawanEURASIP J. Audio Speech Music. Process 2015 · DOI · OpenReview

Introduction

Building high-quality Automatic Speech Recognition (ASR) systems typically requires massive amounts of manually transcribed audio. This requirement poses a significant bottleneck for low-resource languages or specialized domains, such as parliamentary proceedings, where data is scarce and expensive to produce. This paper investigates strategies for leveraging "foreign" data—resources from different languages or domains—to improve ASR performance in a target scenario with limited data.

The researchers specifically address a practical dilemma: given a limited target dataset, is it better to use auxiliary data that shares the same language but a different domain, or data that shares the same domain but a different language?

Methods and Architectures

The study evaluates three primary acoustic modeling architectures to determine their effectiveness in cross-resource scenarios:

  1. HMM/GMM (Baseline): A traditional Hidden Markov Model/Gaussian Mixture Model system using Mel-frequency cepstral coefficients (MFCCs).
  2. HMM/DNN (Hybrid): A Deep Neural Network (DNN) that directly estimates the posterior probabilities of HMM states.
  3. BN-HMM/GMM (Bottleneck): A DNN used as a discriminative feature extractor. A narrow "bottleneck" layer in the middle of the network produces compact features, which are then used as inputs for a traditional HMM/GMM system.

To optimize these models, several key techniques were employed:

  • DNN Adaptation: To prevent overfitting to foreign data, a "condition-specific" layer approach was used. The network was pre-trained on foreign data, after which the output layer was replaced and the entire network was fine-tuned on the target data.
  • Reverberation Robustness: The target parliamentary data was recorded in a large, reverberant chamber. To mitigate temporal masking, the authors integrated Frequency Domain Linear Prediction (FDLP) features, which offer greater robustness to echo than standard MFCCs.
  • Semi-supervised Training: The researchers also explored a two-pass system where an initial model trained on limited data was used to automatically label untranscribed foreign data, expanding the training set.

Datasets

The primary evaluation utilized the MediaParl corpus, a bilingual (French/German) database of proceedings from the Valais parliament in Switzerland. The target dataset consisted of 19 hours of French parliamentary speech. Auxiliary resources included:

  • In-domain, Out-of-language: 18 hours of German parliamentary speech from MediaParl, recorded in the same acoustic environment.
  • Out-of-domain, In-language:
    • ESTER: 57 hours of French radio broadcast news.
  • BREF: 114 hours of French read speech.

Key Insights

The experimental results yielded several critical insights into the behavior of DNN-based ASR systems when using external resources:

Language vs. Domain

A major finding was that linguistic alignment is more critical than acoustic environment matching. French radio news (out-of-domain) proved significantly more beneficial for French parliamentary ASR than German parliamentary speech (in-domain), despite the latter sharing the exact same recording conditions. This suggests that the phonetic and linguistic patterns learned by the DNN are more transferable across domains within the same language than across languages within the same domain.

The Bottleneck Advantage

The BN-HMM/GMM architecture consistently outperformed the HMM/DNN hybrid. By utilizing the DNN as a feature extractor for a GMM backend, the system could leverage mature GMM-based adaptation techniques, such as feature-space Maximum Likelihood Linear Regression (fMLLR), which are more challenging to apply directly to pure DNN hybrids.

Diminishing Returns of Data Volume

Increasing the volume of training data did not always lead to better performance. Combining all French datasets (ESTER and BREF) actually resulted in higher error rates than using ESTER alone. The authors attribute this to the speaking style; the spontaneous nature of radio news in ESTER was a closer match to parliamentary debates than the "read" speech found in BREF.

Results

The transition from the standard HMM/GMM baseline (17.1% Word Error Rate - WER) to the Bottleneck DNN system (14.5% WER) provided a substantial initial improvement. Incorporating the ESTER French radio data further reduced the WER to 13.7%.

By combining the optimal architecture (Bottleneck), the most relevant foreign data (ESTER), reverberation-robust features (FDLP), and speaker adaptation (fMLLR), the final system achieved a WER of 12.1%. This represents a 30% relative improvement over the initial baseline.

Conclusion and Future Work

While the study successfully demonstrated the value of foreign resources, it also highlighted areas for further research. Semi-supervised training showed modest gains, suggesting a need for better confidence measures when selecting auto-labeled data. Additionally, while the study focused on cross-entropy training, the authors noted that sequence-discriminative training (e.g., MPE or sMBR) could potentially yield further performance gains. Future work may also explore more complex intra-sentence code-switching scenarios, as the current study focused on language transitions primarily at sentence boundaries.

Summary updated

Abstract

Manual transcription of audio databases for the development of automatic speech recognition (ASR) systems is a costly and time-consuming process. In the context of deriving acoustic models adapted to a specific application, or in low-resource scenarios, it is therefore essential to explore alternatives capable of improving speech recognition results. In this paper, we investigate the relevance of foreign data characteristics, in particular domain and language, when using this data as an auxiliary data source for training ASR acoustic models based on deep neural networks (DNNs). The acoustic models are evaluated on a challenging bilingual database within the scope of the MediaParl project. Experimental results suggest that in-language (but out-of-domain) data is more beneficial than in-domain (but out-of-language) data when employed in either supervised or semi-supervised training of DNNs. The best performing ASR system, an HMM/GMM acoustic model that exploits DNN as a discriminatively trained feature extractor outperforms the best performing HMM/DNN hybrid by about 5 % relative (in terms of WER). An accumulated relative gain with respect to the MFCC-HMM/GMM baseline is about 30 % WER.
BibTeX
@article{motlicek2015exploiting,
  author = {Petr Motlícek and David Imseng and Blaise Potard and Philip N. Garner and Ivan Himawan},
  title = {{Exploiting foreign resources for DNN-based ASR}},
  journal = {EURASIP J. Audio Speech Music. Process.},
  year = {2015},
  doi = {10.1186/s13636-015-0058-5},
}
Figure