Transfer the Linguistic Representations from TTS to Accent Conversion with Non-Parallel Data

Xi Chen, Jiakun Pei, Liumeng Xue, Mingyang Zhang

TL;DR

This paper introduces a novel, non-autoregressive framework for Foreign Accent Conversion (FAC) that operates without the need for parallel data or a reference target speech during inference. By leveraging linguistic representations from a Text-to-Speech (TTS) system and robust features from the Whisper model, the authors successfully "strip" the source accent while preserving the speaker's unique identity.

What Problem Does This Paper Address?

Foreign Accent Conversion (FAC) aims to transform an individual's accented speech into a native-sounding version while keeping their voice recognizable. This is incredibly useful for second-language learners, personalized TTS, and movie dubbing.

However, the field faces two major hurdles:

  1. Data Scarcity: Most high-performing models require "parallel data"—the same sentence spoken by the same person in both a source and target accent. This is extremely difficult and expensive to collect.
  2. Inference Constraints: Many existing "reference-based" models require a sample of the target accent during the conversion process, which limits real-world utility.

Earlier "reference-free" attempts often used autoregressive models, which are notoriously slow and prone to "unstable attention" (skipping or repeating words), especially in multi-speaker scenarios. This paper seeks a fast, stable, and data-efficient alternative.

Methods

The authors propose a non-autoregressive framework that treats the conversion process as a mapping from "accented speech" to "accent-agnostic linguistic representations."

The Architecture

The model consists of three main components:

  • Speech Encoding Block: Uses a feature extractor (comparing standard mel-spectrograms vs. pre-trained Whisper features) and a Transformer-based encoder to extract content.
  • Textual Auxiliary Block: Based on FastSpeech2, this block is used only during training. It generates "pure" linguistic hidden states from text, serving as the gold standard for what accent-free content should look like.
  • Decoder: A Transformer-based module that takes the speech representations, adds Speaker Embeddings and pitch/energy predictions, and synthesizes the final native-accented mel-spectrogram.

The Three-Stage Training Strategy

  1. Stage 1 (Learning TTS): The model is trained on native speech data to learn how to synthesize high-quality audio from text.
  2. Stage 2 (Speech-Text Alignment): The authors freeze the decoder and optimize the speech encoder to ensure its output matches the linguistic representations from the text encoder. This teaches the encoder to ignore acoustic "noise" like accent.
  3. Stage 3 (Fine-tuning): The model is fine-tuned on accented data (e.g., Hindi-accented English) to adapt the encoder to handle non-native pronunciations while still mapping them to the same "native" linguistic space.

Datasets

  • LibriTTSR: A restored version of the LibriTTS corpus (585 hours, ~2,400 speakers) used for the initial native TTS and alignment stages.
  • L2-ARCTIC: Used for the accent conversion fine-tuning. The authors focused on four Hindi speakers (two male, two female).
  • HiFi-GAN: Used as the vocoder to transform the model's mel-spectrogram outputs into listenable waveforms.

Key Insights and Ideas

  • TTS as a Teacher: The most clever part of this work is using a TTS text-encoder as a "ground truth" for linguistic content. Since text has no accent, forcing the speech encoder to match the text encoder's latent space naturally strips away the source accent.
  • Whisper Features Matter: The authors found that using features from a pre-trained Whisper model (an ASR model) significantly outperformed raw mel-spectrograms. Because Whisper is trained to recognize what is being said across many conditions, its features are inherently more robust to accent variations.
  • Decoupling Content and Identity: By injecting speaker embeddings only at the decoder stage, the model ensures that the "content" extracted by the encoder remains identity-neutral, while the decoder adds the specific "flavor" of the original speaker's voice back in.

Key Results

The proposed method was evaluated against a baseline and several ablation versions:

  • Intelligibility: The model achieved a Word Error Rate (WER) of 17.72%, a massive improvement over the baseline's 24.59%.
  • Audio Quality: In subjective Mean Opinion Score (MOS) tests, the model scored 3.71, outperforming the baseline (3.61) and approaching the "upper bound" of pure TTS (3.86).
  • Accent Reduction: On a 5-point "Accentness" scale (where 1 is "No Accent"), the model scored 1.90, significantly lower than the original Hindi-accented speech (3.49).
  • Identity Preservation: Preference tests showed that listeners found the converted speech's speaker identity to be highly consistent with the original source.

Limitations and Future Work

  • Source Accent Variety: The paper primarily demonstrates results on Hindi-accented English. It remains to be seen how well the "Stage 3" fine-tuning scales to a wider variety of source accents simultaneously.
  • The TTS Gap: While the results are competitive, there is still a small gap in quality between the converted speech and pure TTS output.
  • Prosody Transfer: While the model successfully modifies pronunciation, the authors note that adjusting intonation and rhythm (prosody) to perfectly match native patterns is an ongoing challenge in the FAC field.

Summary updated

Abstract

Accent conversion aims to convert the accent of a source speech to a target accent, meanwhile preserving the speaker’s identity. This paper introduces a novel non-autoregressive framework for accent conversion that learns accent-agnostic linguistic representations and employs them to convert the accent in the source speech. Specifically, the proposed system aligns speech representations with linguistic representations obtained from Text-to-Speech (TTS) systems, enabling training of the accent voice conversion model on non-parallel data. Furthermore, we investigate the effectiveness of a pretraining strategy on native data and different acoustic features within our proposed framework. We conduct a comprehensive evaluation using both subjective and objective metrics to assess the performance of our approach. The evaluation results highlight the benefits of the pretraining strategy and the incorporation of richer semantic features, resulting in significantly enhanced audio quality and intelligibility1.
BibTeX
@inproceedings{chen2024transfer,
  author = {Xi Chen and Jiakun Pei and Liumeng Xue and Mingyang Zhang},
  title = {{Transfer the Linguistic Representations from TTS to Accent Conversion with Non-Parallel Data}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=daoEMriWqF},
}

Loading related content...

Figure