Accent Conversion as Unpaired Translation in Articulatory Phone Space

Foreign accent conversion makes a non-native speaker sound native while keeping their voice. Today's systems buy this with some form of supervision. Some need recordings of native and non-native speakers reading the same sentences. Some need a native reference recording at the moment of conversion. Some need a speech recognizer trained on transcribed data in the target language. We propose to treat accent conversion the way unpaired image-to-image translation treats horses and zebras: learn a mapping between two unaligned collections of speech, one from native and one from non-native speakers. The key change is where the translation happens. It runs over a language-universal phonetic representation, meaning sequences of International Phonetic Alphabet (IPA) symbols, each described by physical articulatory features such as voicing, nasality, and tongue position. It does not run over raw spectrograms.

The central claim is that the standard tool for keeping unpaired translation faithful, cycle consistency, is the wrong constraint for accents. Accent correction is many-to-one: many non-native realizations of "think" should all land on one native pronunciation. It therefore cannot be inverted. Asking a network to reconstruct the accented input from the corrected output invites it to hide the accent inside the output. We replace the round trip with a bounded articulatory edit cost. The converted phone sequence may differ from the input only by moves that are small in articulatory feature space, for example /t/→/θ/ or a vowel shifting in height. Meanwhile, an adversary pushes the sequence toward the statistics of native speech. Both the native and non-native phone streams come from a universal speech-to-IPA recognizer applied to untranscribed audio. The method therefore needs no text, no pronunciation dictionary, and no parallel recordings of the accent pair.

Core Claim and Mechanism

Let $x = (x_1,\dots,x_n)$ be the sequence of phone probability distributions that a frozen universal IPA recognizer produces for a non-native utterance. Let $G$ be a small sequence-to-sequence generator that outputs a converted sequence $G(x)$. Let $\phi(p)$ map a phone $p$ to its articulatory feature vector. We train

$$\mathcal{L}(G) = \mathcal{L}_{adv}\big(G(x);\,P_{L1}\big) + \lambda\,\mathrm{DTW}_{\phi}\big(x, G(x)\big),$$

where $\mathcal{L}_{adv}$ is an adversarial loss from a discriminator that separates $G(x)$ from phone sequences recognized in native (L1) speech, drawn from the distribution $P_{L1}$. $\mathrm{DTW}_{\phi}$ is a soft dynamic-time-warping alignment cost between input and output. Its local cost is the expected distance $\lVert\phi(p)-\phi(q)\rVert_1$ under the two phone distributions, and $\lambda$ sets the edit budget. The warping lets the generator insert or delete short segments, such as epenthetic vowels, which non-native speech often adds or drops. The feature metric makes a near-miss substitution cheap and an arbitrary rewrite expensive. A synthesizer conditioned on IPA then renders $G(x)$ in the source speaker's voice, using the source utterance for speaker identity and timing.

The method draws on three papers in specific roles. The adversary over phone sequences is borrowed from Unsupervised Speech Recognition (wav2vec-U), which matched audio-derived phone sequences to text-derived ones by adversarial training. It depended on a text phonemizer for its "real" samples and ran in one direction only. Our real samples come from recognized native audio, so no phonemizer is needed, and the task is phone-to-phone rather than audio-to-phone. The idea of anchoring translation with a frozen expert instead of a round trip comes from Patch Your Matcher. That paper preserves geometry in cross-sensor image translation by scoring translated images with a frozen feature matcher rather than cycle reconstruction. Our frozen anchor is the IPA recognizer together with the articulatory metric. The feature metric is PanPhon, which maps over 5,000 IPA segments to 21 articulatory features. The recognizer is the zero-shot speech-to-IPA model of Universal Automatic Phonetic Transcription into the IPA, a wav2vec2-XLSR model fine-tuned with CTC over a PanPhon inventory. Both are reused as they are.

Why this gap is real

Three lines of accent conversion work each depend on a different kind of supervision. Zero-Shot Foreign Accent Conversion without a Native Reference translates non-native bottleneck features into native ones with a sequence-to-sequence model. It needs no native reference at inference, but training requires parallel native and non-native recordings of the same prompts. Accent Conversion with Articulatory Representations uses vocal-tract variables and phonetic posteriorgrams, and needs a native reference utterance at inference. USD-AC is unpaired and uses an accent discriminator. It found, however, that accent removal works because its content encoder is a supervised ASR model, and that generic self-supervised features such as HuBERT leak accent. Our proposal removes all three dependencies. That USD-AC finding is also a warning for us: our recognizer is supervised too, only in a language-universal way, trained on about 9 hours across seven languages. Whether a universal IPA recognizer transcribes accented speech faithfully enough is an empirical risk, and we measure it first.

The closest acoustic-level work is CycleGAN-VC3. It carries CycleGAN to non-parallel voice conversion on mel-spectrograms, adding time-frequency adaptive normalization so the spectral structure survives. It has no phonetic intermediate. Linguistic content is preserved only through cyclic spectrogram reconstruction, which is exactly the constraint we argue is ill-posed when the target mapping is many-to-one.

The phonetic-representation literature also constrains the design. One Form to Transfer Them All found that automatic phonemizers inject noise and that romanization beat IPA for text language models. We therefore do not claim IPA is a better text representation. We use it because accents are defined by pronunciation, which orthography cannot express. PanPhon and the universal transcriber both omit lexical tone, so the first study targets non-tonal accent pairs. Tonal dialects are an explicit extension.

Research Program

The first stage is a fidelity audit of the anchor. We run the universal IPA recognizer on non-native and native English speech and ask whether the known substitutions of each first-language group appear in its output, or whether the recognizer quietly normalizes toward canonical phones. Two measurements answer this: phone feature error rate against manually annotated non-native phones where available, and the divergence between the phone-bigram statistics of accent groups. If the recognizer normalizes accents away, the translation has nothing to act on and the project must switch to recognizer outputs from earlier layers.

The second stage builds the generator and discriminator over segment-level phone distributions. As a sanity check, the adversarial component is first run in the wav2vec-U configuration on TIMIT.

The third stage trains the full objective on unpaired native and non-native pools and sweeps $\lambda$ to trace the curve of accentedness against content drift.

The fourth stage renders outputs with a synthesizer that takes IPA input, such as a Koel-TTS-style model over a universal IPA vocabulary as used in Align2Speak. It is conditioned on the source speaker and on source durations. The synthesizer is trained on paired data in high-resource languages. Our "unpaired" claim concerns the accent pair, not speech synthesis in general. The synthesizer is held fixed across all conditions so that differences come from the phone translation.

Baselines

The naive reference point is a copy-synthesis pass. We transcribe the non-native utterance with the universal recognizer and resynthesize it unchanged through the same synthesizer. This separates the effect of passing speech through the synthesizer from any accent change. A system beats it only if native-listener accentedness ratings improve while speaker similarity stays within a small margin.

The strongest phonetic baseline without learning is nearest-native snapping. Each recognized phone outside the native inventory is replaced by the native phone closest in PanPhon distance. This is the "just do lookup" competitor. It uses the same recognizer, metric, and synthesizer, and needs no training.

The strongest existing acoustic-level unpaired method is CycleGAN-VC3, trained between native and non-native pools of matched size, tuned with the same budget of search trials as our generator, and vocoded identically. We found no reported accent-conversion result for it on Lacuna. Its reported evaluation is voice conversion on the Voice Conversion Challenge 2018 English data with 81 training utterances per speaker.

The strongest unpaired accent-specific system is USD-AC. It is fair to run in two forms: with its supervised ASR content encoder, which acts as an upper reference with more supervision than ours, and with HuBERT content, which matches our supervision level. No reported result for USD-AC, the zero-shot bottleneck-translator system, or the articulatory-representation system appears on Lacuna, so all are rerun on our splits.

The only reported number in the file that bears on our components is the TIMIT phone error rate of 11.3 for wav2vec-U's adversarial phone matching, reported in Unsupervised Speech Recognition. Reproducing roughly this figure is our check that the adversary is implemented correctly. It is not an accent-conversion target.

Three ablations isolate the key ingredient. The first replaces the articulatory metric in the alignment cost with plain phone-identity cost, so all substitutions cost the same. The second replaces the edit cost with cycle consistency through a second generator mapping native to non-native. The third removes the adversary, which leaves only snapping-like behavior. The proposal is supported only if the articulatory version beats both the one-hot and cycle versions at matched content drift.

Why not just…?

Why not just transcribe with a native-language ASR and resynthesize from text in the speaker's voice? For English this is a serious threat and may win on intelligibility. It is included as a text-cascade upper reference. It fails in two ways we can measure. First, recognition errors on heavily accented speech change words, whereas our method only changes phones by bounded articulatory steps. We will report word substitutions introduced relative to the source transcript. Second, it requires a written language with an ASR system. The dialect extension, with target dialects of the kind DiaMoE-TTS handles through IPA, is where the cascade is unavailable. We scope our claim to that regime if the cascade dominates on English.

Why not just snap each phone to its nearest native phone? If snapping matches the full method, the adversary is unnecessary, and we will say so. We expect it to fall short because accents are context-dependent and phonotactic. Examples include vowel reduction in unstressed syllables, cluster simplification, and epenthesis. Per-phone lookup cannot express these, while the adversary sees native sequence statistics. The comparison is made at matched feature-edit cost, split by error type.

Why not just keep cycle consistency, which works for images? Cycle consistency assumes the mapping can be inverted. The survey Generative Adversarial Networks: Recent Developments describes it as enforcing round-trip reconstruction, and the known failure for lossy mappings is hidden information carried in the output. The cycle ablation tests this directly. We perturb the cycle model's converted phone posteriors with small noise and check whether back-reconstruction collapses. We also train a probe to predict the speaker's first language from converted outputs. A cycle model that hides the accent should leak it to the probe more than the edit-cost model does.

Why not just use USD-AC with self-supervised features? USD-AC's own finding was that HuBERT content retains accent. Our method differs by working over discrete, interpretable phones with an explicit edit budget. The matched-supervision rerun settles whether this matters.

Proposed Experiments

The main experiments use non-native and native English speech. Native speech comes from LibriSpeech, and TIMIT is used for recognizer auditing. Non-native speech requires a corpus of labeled first-language groups. The file lists none, so we will add one and report no Lacuna reference numbers for it. Speakers are held out between training and test.

Converted speech is scored on five measures. Accentedness is judged by a first-language classifier and by native-listener ratings. Content drift is the phone feature error rate between source and converted phones. Intelligibility is measured by native ASR word error rate. Speaker similarity uses a speaker-verification model, Titanet as in Align2Speak. Naturalness is judged by listeners.

The decisive result is a curve of accentedness reduction against content drift, traced by sweeping $\lambda$. The articulatory edit-cost model must dominate snapping, the one-hot cost, and the cycle version, and must equal or beat CycleGAN-VC3 and HuBERT-based USD-AC at matched speaker similarity. The earlier fidelity audit is a precondition. If the universal recognizer's outputs do not separate accent groups, the claim fails before training begins.

A secondary check moves to a non-tonal dialect pair that lacks standard orthography. There the text cascade cannot be run, and the question is whether the gains hold when the method is the only text-free option.