Abstract
Automatic speech recognition (ASR) systems often struggle with non-native accents due to unique phonetic features and a lack of diverse training data. This research proposes a neural style transfer-based pipeline that modifies non-native speech to resemble native speech in real-time. By treating accent modification as an image-to-image translation problem on spectrograms, the authors achieved up to a 40% relative improvement in ASR accuracy, providing a flexible "wrapper" for existing recognition systems.
Introduction
While modern ASR systems achieve high accuracy for native speakers, their performance drops significantly when processing non-native speech. This "accent gap" stems from the phonetic influence of a speaker's mother tongue and the scarcity of labeled non-native datasets required to train robust models from scratch. This study addresses these challenges by introducing a pre-processing stage that "fixes" the accent before it reaches the ASR system.
Methodology
The core innovation lies in treating speech modification as a visual task. Audio signals are converted into 2D spectrograms, which are then processed by a modification engine.
The researchers explored two primary neural architectures for this task:
- CNN-based Autoencoder: A supervised model trained to map non-native spectrograms directly to native counterparts.
- Neural Style Transfer (NST): A more flexible approach that separates "content" (the spoken words) from "style" (the accent). By applying a native "style matrix" to a non-native "content matrix," the system synthesizes a modified spectrogram that retains the original message but adopts native acoustic characteristics.
Experimental Setup
The system was evaluated using the UME-ERJ dataset, which contains paired English sentences spoken by Japanese students and native Americans. The LibriSpeech dataset was used to train the baseline ASR models and the Loss Network for the style transfer process.
Results and Discussion
The Neural Style Transfer approach consistently outperformed the standard autoencoder. Key findings include:
- Custom ASR Improvement: The Character Error Rate (CER) dropped from 46.3% to 31.7%, a 32% relative improvement.
- Commercial ASR Integration: When used as a wrapper for Google Cloud Speech-to-Text, the CER for non-native speakers fell from 39.8% to 23.9%, a 40% relative improvement.
- Phonetic Alignment: Mean Similarity Scores confirmed that the modified audio was phonetically much closer to native speech.
Conclusion
This research demonstrates that neural style transfer can effectively bridge the accent gap in ASR. By acting as a real-time pre-processing wrapper, this methodology overcomes data scarcity and enhances the accessibility of voice-activated technologies for non-native speakers worldwide.
Summary updated