TL;DR
This paper introduces a novel "Phone Aware Nearest Neighbor" (STM+NN) technique for non-parallel voice conversion (VC). By leveraging a Spectral Transition Measure (STM) to automatically identify phone boundaries without requiring prior training data, the method significantly improves phonetic accuracy and, consequently, the quality and speaker similarity of converted speech compared to existing nearest neighbor alignment methods.
What Problem Does This Paper Address?
Voice Conversion (VC) aims to transform speech from a source speaker to sound like a target speaker. A crucial step in many VC techniques, especially those that don't rely on complex generative models, is aligning the acoustic features of speech from different speakers when they haven't uttered the exact same sentences (non-parallel data). Traditional Nearest Neighbor (NN) based alignment methods, while popular, often suffer from low phonetic accuracy. This is largely because accurately identifying the boundaries of speech sounds (phonemes) is challenging due to the continuous and overlapping nature of speech. While Hidden Markov Models (HMMs) can estimate these boundaries if text transcriptions are available, this requires extensive training data, which is often impractical for target speakers in real-world VC scenarios. This paper addresses the need for an effective, data-efficient method to align non-parallel speech by accurately identifying phone boundaries.
Methods
The core innovation of this paper is the Spectral Transition Measure (STM) based phone boundary estimation. Inspired by how the human auditory cortex responds more strongly to changes than to steady states, STM exploits the rapid variations in spectral features (specifically, Mel Frequency Cepstral Coefficients or MFCCs) to pinpoint locations of significant spectral change. These changes are hypothesized to correspond to phone boundaries.
The process involves:
- Silence Removal: Pre-processing the speech signal.
- MFCC Extraction: Computing 10-dimensional MFCCs with a 30ms window and 10ms frame rate.
- STM Calculation: Using linear regression coefficients of the MFCCs to compute the STM value for each frame. High STM values indicate rapid spectral transitions.
- Peak Detection: Identifying peaks in the STM contour to estimate potential phone boundaries.
- Boundary Adjustment: An algorithm is employed to ensure the number of estimated boundaries matches the number of ground truth boundaries, handling insertions and deletions.
- Phone-Aware Nearest Neighbor (NN) Alignment: Once phone boundaries are estimated for both source and target speakers, the NN technique is applied within corresponding phone segments. This ensures that features from the same phonetic category are aligned, leading to more accurate correspondences.
- Mapping Function Training: The aligned pairs are then used to train a mapping function for the VC system.
The paper contrasts this with a baseline method, likely referring to an iterative NN-based approach like INCA (Iterative Combination of a Nearest Neighbor search and a Conversion step Alignment), which doesn't explicitly use phone boundary information for alignment.
Datasets
The paper uses the CMU-ARCTIC database for experiments. This database contains speech from multiple speakers, allowing for the evaluation of different speaker pairs (male-male, female-female, male-female). For evaluating phonetic accuracy, the TIMIT database is also mentioned, with reference phone annotations obtained from trained annotators or speaker-dependent HMM models. No new benchmarks are introduced; the focus is on improving an existing VC pipeline.
Key Insights and Ideas
- Spectral Transitions as Phone Boundaries: The fundamental idea that rapid spectral changes directly correlate with phonetic boundaries is a powerful and intuitive concept, especially when considering the biological inspiration.
- Data Efficiency: The STM approach bypasses the need for large amounts of transcribed training data, making it highly practical for real-world VC applications where such data is scarce.
- Vowel Importance: The paper highlights that accurate alignment of vowel sounds, which are often characterized by steady-state spectra but crucial for speech quality, is significantly improved by the STM+NN method. This suggests that even subtle spectral transitions within vowels are captured.
- Alignment Quality Directly Impacts VC Performance: The study strongly reinforces the notion that the accuracy of the alignment step is a primary determinant of the final voice conversion quality, impacting both speech naturalness and speaker similarity.
Key Results
- Phonetic Accuracy (PA): The proposed STM+NN technique achieved an average of 13.67% relative improvement in phonetic accuracy compared to the baseline NN-based alignment technique across various speaker pairs.
- Vowel PA: For vowel sounds specifically, the improvement was even more pronounced, with an average of 44.36% absolute increment in correct vowel detection.
- Subjective Evaluation (MOS): The STM+NN based VC systems achieved a 0.175 margin of error (at 95% confidence interval) higher Mean Opinion Scores (MOS) for speech quality compared to the baseline.
- Subjective Evaluation (XAB - Speaker Similarity): In speaker similarity tests, the proposed method was 3.45% more preferred by subjects, with a significant portion (54.05%) giving equal preference, indicating comparable or better performance.
- Computational Efficiency: The proposed alignment algorithm is noted to be faster than the baseline due to avoiding iterative computations.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while the STM algorithm is computationally simple, the exact number of estimated boundaries might not always perfectly match the ground truth, necessitating an adjustment step.
- The evaluation of phonetic accuracy for the CMU-ARCTIC database relies on HMM-derived ground truth, which might have its own limitations compared to manual annotations.
- Future Work:
- The authors express interest in extending this work to cross-lingual voice conversion, which presents even greater challenges in terms of phonetic and linguistic differences.
Summary updated