TL;DR
Researchers from the University of Tokyo have developed a way to make high-quality voice conversion (VC) significantly faster and more efficient. By jointly training a neural network with a "trainable lifter," they reduced the computational complexity of audio filters by 16x without losing speech quality, while a new sub-band processing technique enables high-fidelity 48 kHz voice conversion.
What Problem Does This Paper Address?
Voice conversion—changing a source speaker’s voice to sound like a target speaker while keeping the words the same—is computationally expensive. While deep learning has made voices sound more natural, running these models in real-time on low-power devices like smartphones remains a challenge.
One efficient approach is spectral-differential VC, which modifies the source waveform directly using a digital filter. However, this method typically uses "minimum-phase filters" which require a high number of "taps" (essentially, the filter's memory length). If you simply cut these filters short to save power (truncation), the audio quality plummets. Additionally, when moving from standard 16 kHz audio to high-definition 48 kHz "full-band" audio, traditional models struggle to handle the complex fluctuations in high-frequency sounds, leading to buzzy or distorted results.
Methods
The authors propose two primary innovations:
-
Lifter Training with Filter Truncation: In digital signal processing, a "lifter" helps determine the phase of a filter. Usually, this is a fixed mathematical transform (the Hilbert transform). The authors replaced this fixed component with a trainable lifter. By including the filter truncation process directly in the neural network's training loop (via backpropagation), the model learns to produce a filter that is optimized to sound good even when it is very short. This moves the system from a "minimum-phase" constraint to a "data-driven phase."
-
Sub-band Processing for Full-band VC: To handle 48 kHz audio, the researchers realized that converting the entire frequency range often introduces errors. Their method splits the audio: they apply the conversion filter only to the lower frequencies (below 8 kHz) where the core "identity" of the voice resides, while preserving the original high-frequency components. They use a smooth sigmoid-based transition to blend these bands seamlessly.
Datasets
The study utilized three Japanese speech corpora:
- JSUT corpus: Used for female-to-female (f2f) conversion tasks.
- Voice Actress Corpus: Also used for female-to-female conversion.
- JVS corpus: Used for male-to-male (m2m) conversion tasks. The models were trained on approximately 12 minutes of speech (80 utterances) per speaker.
Key Insights and Ideas
- End-to-End Efficiency: The most significant "aha" moment is that hardware efficiency (filter length) can be treated as a learnable parameter. Instead of designing a filter and then trying to compress it, the model learns to be "compressible" from the start.
- Phase Flexibility: By abandoning the rigid "minimum-phase" requirement, the model finds a phase configuration that concentrates the filter's power into a much smaller window (the first 100 taps), allowing for massive truncation.
- Less is More in Full-band: In high-definition audio, trying to model every tiny fluctuation in the high frequencies often does more harm than good. Preserving the original high-frequency "air" while only converting the lower "vocal" frequencies results in a much more natural-sounding voice.
Key Results
- 16x Speedup: For narrow-band (16 kHz) audio, the proposed method reduced the filter tap length from 512 to just 32. Subjective listening tests showed that this 1/16th-size filter performed just as well as the full-sized conventional version.
- Higher Accuracy: The proposed lifter-training method achieved lower Root Mean Squared Error (RMSE) across all tested filter lengths compared to the conventional approach.
- Superior Full-band Quality: In 48 kHz tests, the sub-band processing method significantly outperformed conventional full-band conversion in speech quality, receiving preference scores of over 0.70 (where 0.50 is a tie).
Limitations and Future Work
- Full-band Truncation Limits: While the method worked perfectly for 16 kHz audio, reducing the filter length for 48 kHz audio still caused some slight degradation in speaker similarity, suggesting that full-band audio requires more complex "memory" in its filters.
- Real-world Latency: While the paper proves the computational efficiency, the authors note that they still need to implement this in a real-time system to measure actual end-to-end latency on mobile hardware.
- Broader Applications: The authors suggest this "trainable lifter" approach could be applied beyond voice conversion to tasks like speech enhancement and source separation.
Summary updated