Personalized Acoustic Modeling by Weakly Supervised Multi-Task Deep Learning using Acoustic Tokens Discovered from Unlabeled Data

Cheng-Kuang Wei, Cheng-Tao Chung, Hung-yi Lee, Lin-Shan LeeICASSP 2017 · DOI · OpenReview

Abstract

Personalized acoustic models are significantly more effective than user-independent ones, but the requirement for transcribed personal data is a major bottleneck. We propose a "weakly supervised" multi-task deep learning framework, the Phoneme-Token Deep Neural Network (PTDNN), which leverages unsupervised acoustic tokens discovered from unlabeled data alongside limited transcribed data. By jointly training on both phonemes and discovered tokens, the model captures speaker-specific nuances more effectively. Experiments on a Facebook post corpus show substantial improvements in word accuracy over standard adaptation techniques like fDLR and speaker codes.

Introduction

Personalizing speech recognition models to individual users is a well-established method for improving performance. However, while collecting audio from mobile devices is easy, manual transcription is costly and slow. Traditional adaptation methods, such as feature-discriminative linear regression (fDLR) or speaker codes, typically require labeled data. Lightly supervised methods use noisy transcriptions from existing models, which can propagate errors. This research explores whether useful linguistic information can be extracted directly from unlabeled data without prior transcription, using a multi-task learning approach.

Methodology

The proposed framework centers on the Phoneme-Token Deep Neural Network (PTDNN). This architecture utilizes shared hidden layers to learn a robust acoustic representation, which then branches into two distinct output heads.

Acoustic Token Discovery

The system first processes unlabeled audio to automatically discover "acoustic tokens." This is achieved through unsupervised segmentation and clustering (e.g., K-means and HMM training). These tokens represent recurring acoustic patterns unique to the user, acting as a "pseudo-language" that captures the speaker's specific phonetic characteristics.

Multi-Task Learning and Adaptation

The PTDNN is trained using a multi-task objective:

  1. Phoneme Head: Trained on a small set of transcribed data to predict standard phonemes.
  2. Token Head: Trained on the large pool of unlabeled data to predict the discovered acoustic tokens.
Overview of the PTDNN architecture and various adaptation strategies: (a) base model, (b) SI model adaptation, (c) supervised adaptation, and (d) the proposed weakly supervised adaptation.

As shown in the figure above, the model can be adapted in several ways. The proposed weakly supervised approach (d) utilizes unlabeled data to adjust the fDLR transform and the token output network, while the phoneme head is fine-tuned using the limited transcribed data.

Experimental Setup

The framework was evaluated using a custom corpus of Facebook posts recorded by 20 speakers (10 male, 10 female). Each speaker provided 1,000 utterances of spontaneous speech. In the "weakly supervised" setup, only 50 utterances per user were transcribed, while 450 utterances remained unlabeled for token discovery. The speech was primarily Mandarin Chinese with a small percentage of English words.

Results and Discussion

The PTDNN approach demonstrated significant gains over traditional baselines. In the scenario with only 50 labeled utterances, the PTDNN achieved a word accuracy of 66.94%, compared to 57.45% for the speaker-independent baseline and 65.70% for standard fDLR adaptation.

Method Word Accuracy (%)
Baseline (Speaker Independent) 57.45
fDLR (Standard Adaptation) 65.70
PTDNN (Proposed) 66.94
PTDNN + fDLR (Combined) 69.83

A key finding is that PTDNN and fDLR are complementary; combining them yielded the best overall performance (69.83%). Furthermore, PTDNN proved highly robust in extreme data-scarcity scenarios, outperforming fDLR by nearly 3% when only 10 transcribed utterances were available.

Conclusion

This work demonstrates that unsupervised acoustic tokens discovered from unlabeled data provide a powerful signal for personalizing acoustic models. By treating token prediction as a secondary task in a multi-task network, the PTDNN effectively transfers knowledge from unlabeled data to the primary phoneme recognition task. Future work will investigate more advanced unsupervised discovery methods and the computational efficiency of the pipeline for on-device applications.

Summary updated

Abstract

It is well known that recognizers personalized to each user are much more effective than user-independent recognizers. With the popularity of smartphones today, although it is not difficult to collect a large set of audio data for each user, it is difficult to transcribe it. However, it is now possible to automatically discover acoustic tokens from unlabeled personal data in an unsupervised way. We therefore propose a multi-task deep learning framework called a phoneme-token deep neural network (PTDNN), jointly trained from unsupervised acoustic tokens discovered from unlabeled data and very limited transcribed data for personalized acoustic modeling. We term this scenario "weakly supervised". The underlying intuition is that the high degree of similarity between the HMM states of acoustic token models and phoneme models may help them learn from each other in this multi-task learning framework. Initial experiments performed over a personalized audio data set recorded from Facebook posts demonstrated that very good improvements can be achieved in both frame accuracy and word accuracy over popularly-considered baselines such as fDLR, speaker code and lightly supervised adaptation. This approach complements existing speaker adaptation approaches and can be used jointly with such techniques to yield improved results.
BibTeX
@inproceedings{wei2017personalized,
  author = {Cheng-Kuang Wei and Cheng-Tao Chung and Hung-yi Lee and Lin-Shan Lee},
  title = {{Personalized Acoustic Modeling by Weakly Supervised Multi-Task Deep Learning using Acoustic Tokens Discovered from Unlabeled Data}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  doi = {10.1109/icassp.2017.7953141},
}
Figure