Abstract
Accent conversion aims to convert the accent of a source speech to a target accent, meanwhile preserving the speaker's identity. This paper introduces a novel non-autoregressive framework for accent conversion that learns accent-agnostic linguistic representations and employs them to convert the accent in the source speech. Specifically, the proposed system aligns speech representations with linguistic representations obtained from Text-to-Speech (TTS) systems, enabling training of the accent voice conversion model on non-parallel data. Furthermore, we investigate the effectiveness of a pretraining strategy on native data and different acoustic features within our proposed framework. We conduct a comprehensive evaluation using both subjective and objective metrics to assess the performance of our approach. The evaluation results highlight the benefits of the pretraining strategy and the incorporation of richer semantic features, resulting in significantly enhanced audio quality and intelligibility.
Introduction
Foreign Accent Conversion (FAC) is the process of transforming an individual's accented speech into a native-sounding version while maintaining their unique vocal identity. This technology holds significant potential for language learning, personalized speech synthesis, and entertainment dubbing. However, two primary challenges have hindered progress in this field: the scarcity of parallel data (the same sentence spoken by both native and non-native speakers) and the instability of autoregressive models, which often suffer from word skipping or repetition.
This paper presents a novel, non-autoregressive framework designed to overcome these hurdles. By leveraging linguistic representations from a Text-to-Speech (TTS) system and robust features from the Whisper model, the proposed method achieves high-quality accent conversion without the need for parallel data or a reference target speech during inference.
Proposed Framework
The architecture treats accent conversion as a mapping from accented speech to accent-agnostic linguistic representations. The system is composed of three primary modules:
- Speech Encoding Block: This module extracts content from the input audio. The authors compared standard mel-spectrograms against pre-trained Whisper features, finding that the latter provides a more robust semantic foundation for the conversion process.
- Textual Auxiliary Block: Based on the FastSpeech2 architecture, this block is utilized exclusively during the training phase. It generates "pure" linguistic hidden states from text, serving as a ground-truth reference for what accent-free content should represent.
- Decoder: A Transformer-based module that synthesizes the final native-accented mel-spectrogram. It integrates the speech representations with speaker embeddings and predicted pitch/energy contours to ensure the output retains the original speaker's identity.
Training Strategy
To bridge the gap between accented speech and native linguistic representations, the authors employ a three-stage training strategy:
- Stage 1: Native TTS Pretraining: The model is first trained on a large corpus of native speech (LibriTTSR) to learn the fundamentals of high-quality audio synthesis from text.
- Stage 2: Speech-Text Alignment: The decoder is frozen, and the speech encoder is optimized to ensure its output aligns with the linguistic representations generated by the text encoder. This process effectively teaches the encoder to ignore acoustic "noise" such as accent and focus on core linguistic content.
- Stage 3: Fine-tuning: The model is fine-tuned on accented data (specifically Hindi-accented English from the L2-ARCTIC dataset). This allows the encoder to adapt to non-native pronunciations while mapping them into the established native linguistic space.
Experimental Results
The framework was evaluated using both objective and subjective metrics, comparing it against baseline models and various ablation studies.
Intelligibility and Quality
The proposed model achieved a Word Error Rate (WER) of 17.72%, a significant improvement over the baseline's 24.59%. In subjective Mean Opinion Score (MOS) tests, the model scored 3.71, outperforming the baseline (3.61) and nearing the performance of pure TTS systems (3.86).
Accent Reduction
On a 5-point "Accentness" scale (where 1 represents no accent), the model's output was rated at 1.90. This marks a substantial reduction from the original Hindi-accented speech, which was rated at 3.49. Furthermore, preference tests confirmed that the model successfully preserved the source speaker's identity throughout the conversion.
Key Insights
- TTS as a Teacher: Using a TTS text-encoder as a reference for linguistic content is a powerful way to "strip" accents. Because text is inherently accent-free, forcing the speech encoder to match this latent space naturally filters out non-native pronunciation patterns.
- The Power of Whisper: Pre-trained Whisper features proved superior to raw mel-spectrograms. Because Whisper is trained on vast amounts of diverse speech for recognition, its features are inherently more resilient to accent variations.
- Decoupling Content and Identity: By isolating speaker embeddings within the decoder, the model ensures that the linguistic "content" remains identity-neutral, while the decoder adds the specific vocal characteristics of the original speaker back into the synthesized audio.
Conclusion
This research demonstrates that a non-autoregressive framework leveraging TTS-derived linguistic representations can effectively perform accent conversion without parallel data. The combination of Whisper features and a staged training approach significantly improves both the intelligibility and the naturalness of the converted speech, providing a robust path forward for reference-free accent conversion technology.
Summary updated