Accent modification for speech recognition of non-native speakers using neural style transfer

Kacper Radzikowski, Le Wang, Osamu Yoshie, Robert NowakEURASIP 2021 · DOI

Abstract

Automatic speech recognition (ASR) systems often struggle with non-native accents due to unique phonetic features and a lack of diverse training data. This research proposes a neural style transfer-based pipeline that modifies non-native speech to resemble native speech in real-time. By treating accent modification as an image-to-image translation problem on spectrograms, the authors achieved up to a 40% relative improvement in ASR accuracy, providing a flexible "wrapper" for existing recognition systems.

Introduction

While modern ASR systems achieve high accuracy for native speakers, their performance drops significantly when processing non-native speech. This "accent gap" stems from the phonetic influence of a speaker's mother tongue and the scarcity of labeled non-native datasets required to train robust models from scratch. This study addresses these challenges by introducing a pre-processing stage that "fixes" the accent before it reaches the ASR system.

Methodology

The core innovation lies in treating speech modification as a visual task. Audio signals are converted into 2D spectrograms, which are then processed by a modification engine.

The proposed pipeline for accent modification and speech recognition.

The researchers explored two primary neural architectures for this task:

  1. CNN-based Autoencoder: A supervised model trained to map non-native spectrograms directly to native counterparts.
  2. Neural Style Transfer (NST): A more flexible approach that separates "content" (the spoken words) from "style" (the accent). By applying a native "style matrix" to a non-native "content matrix," the system synthesizes a modified spectrogram that retains the original message but adopts native acoustic characteristics.
Schema of the CNN-based autoencoder used for accent modification.

Experimental Setup

The system was evaluated using the UME-ERJ dataset, which contains paired English sentences spoken by Japanese students and native Americans. The LibriSpeech dataset was used to train the baseline ASR models and the Loss Network for the style transfer process.

Results and Discussion

The Neural Style Transfer approach consistently outperformed the standard autoencoder. Key findings include:

  • Custom ASR Improvement: The Character Error Rate (CER) dropped from 46.3% to 31.7%, a 32% relative improvement.
  • Commercial ASR Integration: When used as a wrapper for Google Cloud Speech-to-Text, the CER for non-native speakers fell from 39.8% to 23.9%, a 40% relative improvement.
  • Phonetic Alignment: Mean Similarity Scores confirmed that the modified audio was phonetically much closer to native speech.

Conclusion

This research demonstrates that neural style transfer can effectively bridge the accent gap in ASR. By acting as a real-time pre-processing wrapper, this methodology overcomes data scarcity and enhances the accessibility of voice-activated technologies for non-native speakers worldwide.

Summary updated

Abstract

Abstract Nowadays automatic speech recognition (ASR) systems can achieve higher and higher accuracy rates depending on the methodology applied and datasets used. The rate decreases significantly when the ASR system is being used with a non-native speaker of the language to be recognized. The main reason for this is specific pronunciation and accent features related to the mother tongue of that speaker, which influence the pronunciation. At the same time, an extremely limited volume of labeled non-native speech datasets makes it difficult to train, from the ground up, sufficiently accurate ASR systems for non-native speakers.In this research, we address the problem and its influence on the accuracy of ASR systems, using the style transfer methodology. We designed a pipeline for modifying the speech of a non-native speaker so that it more closely resembles the native speech. This paper covers experiments for accent modification using different setups and different approaches, including neural style transfer and autoencoder. The experiments were conducted on English language pronounced by Japanese speakers ( UME-ERJ dataset). The results show that there is a significant relative improvement in terms of the speech recognition accuracy. Our methodology reduces the necessity of training new algorithms for non-native speech (thus overcoming the obstacle related to the data scarcity) and can be used as a wrapper for any existing ASR system. The modification can be performed in real time, before a sample is passed into the speech recognition system itself.
BibTeX
@inproceedings{radzikowski2021accent,
  author = {Kacper Radzikowski and Le Wang and Osamu Yoshie and Robert Nowak},
  title = {{Accent modification for speech recognition of non-native speakers using neural style transfer}},
  booktitle = {EURASIP},
  year = {2021},
  doi = {10.1186/s13636-021-00199-3},
}
Figure