Abstract
In this paper, we propose computationally efficient and high-quality methods for statistical voice conversion (VC) with direct waveform modification based on spectral differentials. The conventional method with a minimum-phase filter achieves high-quality conversion but requires heavy computation in filtering. This is because the minimum phase using a fixed lifter of the Hilbert transform often results in a long-tap filter. One of our methods is a data-driven method for lifter training. Since this method takes filter truncation into account in training, it can shorten the tap length of the filter while preserving conversion accuracy. Our other method is sub-band processing for extending the conventional method from narrow-band (16 kHz) to full-band (48 kHz) VC, which can convert a full-band waveform with higher converted-speech quality. Experimental results indicate that 1) the proposed lifter-training method for narrow-band VC can shorten the tap length to 1/16 without degrading the converted-speech quality and 2) the proposed sub-band-processing method for full-band VC can improve the converted-speech quality than the conventional method.
Introduction
Voice conversion (VC) technology aims to transform a source speaker's voice to match a target speaker's identity while preserving the original linguistic content. While deep learning has significantly improved the naturalness of converted speech, real-time implementation on resource-constrained devices like smartphones remains a significant challenge due to high computational costs.
One efficient approach is spectral-differential VC, which modifies the source waveform directly using a digital filter. However, traditional implementations typically rely on "minimum-phase filters" which require a high number of "taps" (filter memory length) to maintain audio quality. If these filters are truncated to save power, the audio quality often plummets. Furthermore, extending these methods to high-fidelity 48 kHz "full-band" audio introduces complexities in modeling high-frequency fluctuations, often leading to buzzy or distorted results.
Proposed Methods
The researchers introduce two primary innovations to address these efficiency and quality bottlenecks.
1. Lifter Training with Filter Truncation
In conventional digital signal processing for VC, a "lifter" helps determine the phase of a filter, usually through a fixed mathematical transform known as the Hilbert transform. The proposed method replaces this fixed component with a trainable lifter powered by a deep neural network (DNN).
By incorporating the filter truncation process directly into the neural network's training loop via backpropagation, the model learns to produce a filter optimized for performance even when it is very short. This transition from a rigid "minimum-phase" constraint to a "data-driven phase" allows the model to concentrate the filter's power into a much smaller temporal window.
2. Sub-band Processing for Full-band VC
To handle 48 kHz audio, the researchers developed a sub-band processing approach. Recognizing that converting the entire frequency range can introduce errors in high-frequency modeling, the conversion filter is applied only to the lower frequencies (below 8 kHz) where the core vocal identity resides. The original high-frequency components are preserved and blended with the converted lower band using a smooth sigmoid-based transition, ensuring a natural-sounding "full-band" output.
Experimental Setup
The proposed methods were evaluated using three Japanese speech corpora: the JSUT corpus, the Voice Actress Corpus, and the JVS corpus. These datasets provided a range of female-to-female (f2f) and male-to-male (m2m) conversion tasks. Models were trained on approximately 12 minutes of speech (80 utterances) per speaker.
Results and Discussion
The experimental results highlight significant gains in both efficiency and quality.
Efficiency and Accuracy
The lifter-training method achieved a massive reduction in computational complexity. For narrow-band (16 kHz) audio, the filter tap length was reduced from 512 to just 32—a 16x speedup—without any degradation in subjective speech quality.
As shown in the performance analysis, the proposed method consistently achieves lower Root Mean Squared Error (RMSE) across all tested filter lengths compared to the conventional approach. This demonstrates that hardware efficiency can be treated as a learnable parameter; the model learns to be "compressible" from the start.
Full-band Quality
In 48 kHz tests, the sub-band processing method significantly outperformed conventional full-band conversion. Subjective listening tests yielded preference scores of over 0.70 (where 0.50 represents a tie), indicating that preserving original high-frequency "air" while converting only the vocal frequencies results in superior high-fidelity speech.
Conclusion
This research proves that high-quality voice conversion does not have to be computationally prohibitive. By moving from fixed mathematical constraints to data-driven phase modeling and intelligent sub-band processing, the authors have paved the way for high-fidelity VC on low-power devices. Future work will involve implementing these methods in real-time systems to measure end-to-end latency and exploring applications in related fields such as speech enhancement and source separation.
Summary updated