Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restoration

Shigeki Karita, Yuma Koizumi, Heiga Zen, Haruko Ishikawa, Robin Scheibler, Michiel Bacchiani

TL;DR

Miipher-2 is a highly efficient, universal speech restoration (SR) model designed to clean massive, million-hour scale datasets for training generative models. By leveraging a frozen Universal Speech Model (USM) and parameter-efficient adapters, it achieves state-of-the-art restoration quality across 300+ languages without requiring text or speaker metadata.

Problem Statement

The performance of modern generative audio models, such as Text-to-Speech (TTS) systems, is heavily dependent on the volume and quality of their training data. While the internet provides a nearly bottomless source of speech data, web-scraped audio is often plagued by noise, reverberation, and low-quality codecs.

Existing speech restoration models face three major hurdles when trying to clean data at a "million-hour" scale:

  1. Language Generalization: Most models are monolingual or support only a few high-resource languages.
  2. Conditioning Requirements: Many state-of-the-art models require ground-truth text transcripts or speaker IDs, which are expensive or impossible to obtain for massive web-scraped datasets.
  3. Computational Efficiency: Restoring millions of hours of audio requires extreme throughput and low memory footprints to be economically feasible.

Methods

Miipher-2 introduces a "conditioning-free" architecture that operates directly on noisy waveforms. Its design focuses on three main components:

  • Robust Feature Extraction: The model uses a frozen, 2-billion parameter Universal Speech Model (USM). Because the USM was pre-trained on 12 million hours of diverse, noisy audio across 300+ languages, its internal representations are inherently robust to noise and capture fine-grained acoustic details without needing text or speaker labels.
  • Parameter-Efficient Feature Cleaner: To predict "clean" features from noisy ones, the authors use Parallel Adapters (PAs)—small feed-forward networks appended to the USM layers. This approach reduces the number of trainable parameters from 100M (in the original Miipher) to just 20M, significantly speeding up both training and inference.
  • Memory-Efficient Vocoder: For the final step of turning features back into audio, they employ a modified WaveFit vocoder. They optimized its architecture by replacing memory-heavy transposed convolutions with simple upsampling and simplifying the Feature-wise Linear Modulation (FiLM) layers to reduce the memory footprint on hardware accelerators like TPUs.
The Miipher-2 architecture uses a frozen USM encoder with parallel adapters and a modified WaveFit vocoder.

Datasets

The model was trained on a 3,000-hour multi-lingual dataset spanning 54 languages. This "clean" data was augmented with simulated noise, reverberation, and codec artifacts to create noisy-clean pairs. For evaluation, the authors used LibriTTS to benchmark against the original English-only Miipher, Multilingual LibriSpeech (MLS) for known languages, and FLEURS to evaluate "zero-shot" restoration on languages not seen during the feature cleaner's training.

Key Insights and Results

SSL as a Universal Conditioner

The most striking insight is that a powerful self-supervised learning (SSL) model like USM provides enough acoustic context to obviate the need for text or speaker IDs. The "bottleneck" of needing metadata for high-quality restoration is effectively broken.

Efficiency of Parallel Adapters

The Parallel Adapter approach not only uses fewer parameters but also converges much faster and reaches a lower loss than full fine-tuning of the USM or using a standard Conformer-based cleaner. This efficiency is critical for training on massive datasets.

Comparison of loss curves shows that Parallel Adapters (green) converge faster and to a lower loss than other strategies.

Performance at Scale

Miipher-2 sets a new bar for efficient, large-scale speech restoration:

  • Extreme Speed: It achieves a Real-Time Factor (RTF) of 0.0078. This means 100 consumer-grade accelerators can process one million hours of speech in just three days, representing a 724% speed improvement over their baseline.
  • Superior Quality: In subjective human evaluations (Mean Opinion Score), Miipher-2 significantly outperformed the original Miipher-1 (3.46 vs 3.26).
  • Robust Multilingualism: The model successfully restored audio in "unknown" languages from the FLEURS dataset, showing improved sound quality (DNSMOS) and naturalness (SQuId) across the board.

Limitations and Future Work

The authors acknowledge a few key limitations. While the audio sounds much cleaner, the Word Error Rate (WER) for some low-resource languages remains high, likely due to evaluation model limitations. Additionally, due to safety concerns regarding high-fidelity restoration, the authors have chosen not to release the code or model checkpoints. Future work will explore "dataset distillation"—using Miipher-2 to bootstrap high-quality generative models for languages that currently lack studio-quality corpora.

Summary updated

BibTeX
@inproceedings{karita2025miipher,
  author = {Shigeki Karita and Yuma Koizumi and Heiga Zen and Haruko Ishikawa and Robin Scheibler and Michiel Bacchiani},
  title = {{Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restoration}},
  booktitle = {WASPAA},
  year = {2025},
  doi = {10.1109/waspaa66052.2025.11230923},
}

Loading related content...

Figure