Multi-Lingual Pronunciation Assessment with Unified Phoneme Set and Language-Specific Embeddings

Binghuai Lin, Liyuan WangICASSP 2023 · OpenReview

Abstract

Automatic pronunciation assessment is commonly trained and applied for a specific language, which is not practical in multi-lingual or low-resource scenarios. In this paper, we propose a unified method to take advantage of multi-lingual data for multi-lingual pronunciation assessment. To this end, we first construct a concise unified phoneme set for multi-lingual phoneme recognition based on a pre-trained acoustic model. In this way we can not only share language-independent knowledge but also try to discriminate language-specific information for pronunciation assessment. Second, we employ language-specific embeddings for different languages, which act like language-specific assessment criteria to adaptively adjust the feature weights based on an attention mechanism. The whole network is optimized in a unified framework. Experimental results based on multi-lingual datasets demonstrate its superiority to different baselines in Pearson correlation coefficient (PCC). We also illustrate the generalizability of the proposed method for both seen and unseen data.
BibTeX
@inproceedings{lin2023multi,
  author = {Binghuai Lin and Liyuan Wang},
  title = {{Multi-Lingual Pronunciation Assessment with Unified Phoneme Set and Language-Specific Embeddings}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=HJa9E7yji1},
}

Loading related content...

Figure