Abstract
This paper proposes a method of acoustic modeling for zero-resourced languages speech recognition under mismatch conditions. In those languages, very limited or no transcribed speech is available for traditional monolingual speech recognition. Conventional methods such as IPA based universal acoustic modeling has been proved to be effective under matched acoustic conditions, while usually poorly preformed when mismatch occurs. Since mismatch problems between languages often appears, in this paper, unsupervised acoustic modeling via cross-lingual knowledge sharing has thus been proposed: first, initial acoustic models (AM) for a target zero-resourced language are trained using Multi-Task Deep Neural Networks (MDNN) -different languages' speech mapped to the phonemes of the target language (mapped data) is jointly trained together with the same data transcribed language specifically and respectively (specific data); then, automatically transcribed target language data is used in the iterative process to train new AMs, with various auxiliary tasks. Experiment on 100 hour Japanese speech without transcripts achieved a character error rate (CER) of 57.21%, 19.32% absolute improvement compared to baseline (IPA based universal acoustic modeling).
Introduction
Building Automatic Speech Recognition (ASR) systems for "zero-resource" languages—those lacking transcribed audio data—remains a significant challenge in speech processing. Traditional ASR systems require extensive manually transcribed datasets, which are often unavailable or too costly to collect for many of the world's languages.
Existing approaches often rely on "universal" models using the International Phonetic Alphabet (IPA) to map sounds from a source language to a target language. However, these models frequently fail due to acoustic mismatches in speaking styles or language families. Furthermore, unsupervised learning attempts often suffer from a "garbage in, garbage out" cycle, where errors in the initial seed model propagate through subsequent iterations.
Methodology: Multi-Task Deep Neural Networks
To address these issues, this research introduces a two-stage approach centered on Multi-Task Deep Neural Networks (MDNN).
Initial Seed Modeling
The core of the proposed method is an MDNN architecture with shared hidden layers and multiple output layers representing different tasks. The primary task focuses on the target language (Japanese) phonemes, mapped from source languages. Simultaneously, complementary tasks handle the specific phonemes of the source languages (Mandarin, English, and Korean). This structure enables the model to learn robust, general acoustic features while specializing in the target phoneme set.
Iterative Refinement and Acoustic Stability
Following the creation of a seed model, the system transcribes raw target language audio. To mitigate transcription errors, the researchers employed Acoustic Stability (A-stab). Unlike standard confidence scores, A-stab measures consistency in decoding across different weights between acoustic and language models. During iteration, Context-Independent (CI) phoneme targets were added as auxiliary tasks to stabilize training against noisy data.
Experimental Setup
The system was evaluated using 100 hours of Japanese Conversational Telephone Speech (CTS) without manual transcripts. Source data included 100 hours each of Mandarin (HKUST), English (Fisher), and Korean. A supervised model trained on the same 100 hours of Japanese served as the performance upper bound (42.23% CER).
Key Results and Discussion
The proposed MDNN-based seed model significantly outperformed the standard IPA-based baseline. While the baseline yielded a 76.53% CER, the MDNN seed model achieved 69.83% CER.
The iterative refinement process further enhanced performance. As shown in the figure below, the Character Error Rate (CER) consistently decreased over five iterations, eventually reaching 57.21%.
Insights on Data Selection
The study revealed that data quality is more critical than quantity during iteration. Using the 60 hours of "most stable" data resulted in better performance than using the full 100 hours, emphasizing the importance of filtering out noisy transcriptions in unsupervised loops. Additionally, the researchers found that using a smaller, more focused set of mapped data (100h vs 300h) for the initial seed model was more effective, likely because excessive mismatched data can degrade the shared hidden layers.
Conclusion
The proposed method achieved a total absolute improvement of 19.32% over the baseline, significantly narrowing the gap between zero-resource and supervised systems. While a 57.21% CER indicates that challenges remain for practical applications, the success of cross-lingual knowledge sharing via MDNNs provides a robust framework for future developments in unsupervised acoustic modeling. Future work will explore incorporating adaptation methods to further tune models to specific target language environments.
Summary updated