MimicLM: Zero-Shot Voice Imitation through Autoregressive Modeling of Pseudo-Parallel Speech Corpora

Tao Feng, Yuxiang Wang, Yuancheng Wang, Xueyao Zhang, Dekun Chen, Chaoren Wang, Xun Guan, Zhizheng Wu

TL;DR

MimicLM is an end-to-end zero-shot voice imitation model that achieves human-like naturalness by training on a novel "role-swapped" pseudo-parallel dataset. By using synthetic speech as the source and real recordings as the target, the model breaks the quality ceiling imposed by traditional synthetic-target training. Combined with interleaved text-audio modeling and Direct Preference Optimization (DPO), MimicLM delivers state-of-the-art naturalness while precisely mimicking speaker timbre, accent, and emotion.

What Problem Does This Paper Address?

While traditional voice conversion (VC) focuses purely on transferring a speaker's unique vocal tone (timbre), voice imitation (VI) aims for a much higher bar: replicating the complete speaking style, including prosody, rhythm, and emotional expressiveness.

The core bottleneck in training voice imitation models is data scarcity. Ideally, you need triplets of (source speech, reference voice, target speech), where the source and target share the exact same text, but the target matches the reference speaker's style. Since humans rarely record identical sentences across thousands of different speakers, researchers usually rely on Text-to-Speech (TTS) or VC systems to generate synthetic target data.

However, this introduces a synthetic quality ceiling: the model can only ever be as good as the external generator used to create the training targets. MimicLM addresses this limitation by flipping the script on how synthetic data is utilized.

Methods

MimicLM introduces three key innovations to achieve high-fidelity voice imitation without complex, multi-stage disentanglement architectures:

1. Role-Swapping Data Construction

Instead of converting real speech into imperfect synthetic speech, the authors propose a "role-swapping" strategy. As shown in the pipeline below, they take a real utterance from Speaker A, use a TTS model to re-synthesize that exact text in Speaker B's voice, and then train MimicLM to convert that synthetic speech back into Speaker A's original real recording.

Figure 2: Four-stage pipeline for pseudo-parallel data construction. We randomly sample two speakers with their utterances (Stage 1), synthesize cross-speaker audio using TTS (Stage 2), apply role-swapping to ensure real speech serves as training target (Stage 3), and filter low-quality pairs via ASR-based verification (Stage 4).

Because the target is always a genuine human recording, the model learns directly from real speech distributions, completely bypassing the synthetic quality ceiling.

2. Interleaved Text-Audio Modeling

Simultaneously changing timbre and prosody alters the temporal structure of speech, which often leads to dropped words or mispronunciations (high Word Error Rate). To anchor the content, MimicLM uses a decoder-only Transformer that interleaves text tokens and audio tokens during generation.

By structuring the chunks so that text tokens slightly lead the audio tokens (a 1:5 ratio), the model gains a "lookahead" semantic context that guides highly intelligible speech synthesis.

Figure 1: Overview of MimicLM architecture for voice imitation. The model processes reference audio (ref token) for target timbre/style and source audio (src token), then generates the conversion target in two phases: a chunked phase with interleaved text-audio prediction, followed by a continuous phase for remaining speech tokens. Both the audio tokenizer and decoder are frozen during training (snowflake icons). Special control tokens are annotated at bottom right.

3. Preference Alignment via DPO

Training on synthetic inputs but performing inference on real-world speech creates a "synthetic-to-real" distributional gap. To fix this, the authors apply Direct Preference Optimization (DPO) during post-training. They feed real speech into the model, sample multiple outputs, rank them using automatic Word Error Rate (WER) and similarity metrics, and use these preference pairs to fine-tune the model for real-world deployment.

Datasets

  • Training: The model was trained on a massive scale using the Emilia dataset. After filtering out low-quality audio via an ASR-based quality control step (keeping pairs with WER < 0.1), they yielded 8.5 million English pairs (~18,000 hours) and 0.74 million Chinese pairs (~1,600 hours).
  • Evaluation:
    • SeedTTS test-vc-en / test-zh: Standard zero-shot benchmarks used to evaluate cross-lingual voice conversion and TTS.
  • MimicLM-Test: A newly introduced diagnostic dataset consisting of 833 speaker pairs from Emilia, specifically designed to analyze the synthetic-to-real domain gap.

Key Insights and Ideas

  • The Power of Inversion: The most profound takeaway is that synthetic data is better used as an input rather than an output. Forcing a model to predict real human speech distributions inherently yields cleaner, more natural results than forcing it to mimic a TTS system's artifacts.
  • Semantic Anchoring: Interleaving text tokens directly into the autoregressive audio stream acts as a powerful regularizer. It prevents the content collapse typically seen in style-heavy voice conversion models.
  • DPO Bridges the Domain Gap: DPO isn't just for aligning LLM text responses; it is highly effective at correcting acoustic and intelligibility errors caused by domain mismatch in speech-to-speech models.

Key Results

MimicLM was evaluated against top-tier zero-shot voice conversion and imitation baselines (including Vevo and SeedVC v2):

  • Superior Naturalness: On the SeedTTS benchmark, MimicLM achieved the highest naturalness scores among full voice imitation models, scoring 3.22 on UTMOS and 4.15 on OVRL (DNSMOS), outperforming Vevo (2.83 UTMOS).
  • Intelligibility Boost: The DPO alignment phase dramatically reduced the Word Error Rate (WER) on real inputs from 12.80% (SFT stage) down to 8.25%, effectively bridging the synthetic-to-real gap.
  • Scaling Performance: As shown in the scaling analysis, both intelligibility (WER) and speaker similarity (S-SIM) show consistent, predictable improvements as data scales from 100K to 8.5M samples, indicating further potential with larger datasets.
Impact of training data scale on WER and speaker similarity

Limitations and Future Work

Despite its strong performance, the authors note several limitations:

  1. Upstream TTS Dependence: While the target is real, the source is still synthetic. If the upstream TTS engine produces extremely poor audio, those samples must be filtered out, reducing data efficiency (33% of data was discarded during filtering).
  2. The Intelligibility Tax: MimicLM still exhibits a higher WER than simpler, timbre-only voice conversion systems. Jointly transforming both timbre and prosody remains an inherently complex task.
  3. Resource Intensive: The two-stage training pipeline and large-scale data generation require significant compute resources, which may limit accessibility for resource-constrained researchers.

Summary updated

Abstract

Voice imitation aims to transform source speech to match a reference speaker's timbre and speaking style while preserving linguistic content. A straightforward approach is to train on triplets of (source, reference, target), where source and target share the same content but target matches the reference's voice characteristics, yet such data is extremely scarce. Existing approaches either employ carefully designed disentanglement architectures to bypass this data scarcity or leverage external systems to synthesize pseudo-parallel training data. However, the former requires intricate model design, and the latter faces a quality ceiling when synthetic speech is used as training targets. To address these limitations, we propose MimicLM, which takes a novel approach by using synthetic speech as training sources while retaining real recordings as targets. This design enables the model to learn directly from real speech distributions, breaking the synthetic quality ceiling. Building on this data construction approach, we incorporate interleaved text-audio modeling to guide the generation of content-accurate speech and apply post-training with preference alignment to mitigate the inherent distributional mismatch when training on synthetic data. Experiments demonstrate that MimicLM achieves superior voice imitation quality with a simple yet effective architecture, significantly outperforming existing methods in naturalness while maintaining competitive similarity scores across speaker identity, accent, and emotion dimensions.
BibTeX
@inproceedings{feng2026mimiclm,
  author = {Tao Feng and Yuxiang Wang and Yuancheng Wang and Xueyao Zhang and Dekun Chen and Chaoren Wang and Xun Guan and Zhizheng Wu},
  title = {{MimicLM: Zero-Shot Voice Imitation through Autoregressive Modeling of Pseudo-Parallel Speech Corpora}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1020},
}
Figure