Abstract
The Voice Conversion Challenge 2016 is the first Voice Conversion Challenge in which different voice conversion systems and approaches using the same voice data were compared. This paper describes the design of the evaluation, it presents the results and statistical analyses of the results.
Introduction
Voice Conversion (VC) aims to modify a source speaker's audio to sound like a specific target speaker while preserving linguistic content. Historically, benchmarking VC techniques was difficult due to the lack of standardized datasets and metrics. The inaugural Voice Conversion Challenge (VCC) 2016 established a rigorous benchmark for 17 different systems. This paper analyzes the results, revealing that while modern systems can achieve up to 70% speaker similarity, maintaining high naturalness—especially in cross-gender conversions—remains a significant hurdle.
Methodology
The evaluation framework was designed to handle 18 systems (17 participants and one baseline) across various gender combinations without exhausting human listeners.
Datasets
The challenge utilized a controlled dataset of 5 source and 5 target speakers. From an initial 25 source-target (ST) pairs, 16 were selected for the final evaluation, excluding speakers with irregular rates or shouted speech. Each speaker provided a pool of 30 test sentences (2-5 seconds long).
Perceptual Evaluation
A two-pronged listening test was implemented:
- Naturalness: Listeners rated samples on a 5-point Mean Opinion Score (MOS) scale, from "totally unnatural" (1) to "completely natural" (5).
- Similarity: The authors employed a "same/different" paradigm. Listeners heard pairs of clips and decided if they were the same speaker, also indicating confidence (e.g., "Same: absolutely sure"). This approach better reflects natural human voice recognition compared to standard MOS similarity scales.
Key Results and Discussion
The evaluation revealed a clear performance gap between synthetic systems and natural speech.
Naturalness and Statistical Groupings
Most systems clustered between 2.0 and 3.5 on the MOS scale, significantly lower than the natural human "gold standard" (Systems S and T). Significance groupings showed that many top-performing systems were not statistically distinguishable from one another.
Speaker Similarity
Top systems reached approximately 70% similarity. However, a significant portion of these scores included "not sure" responses. For instance, System J achieved a high "same" rating, but a large percentage of those respondents were uncertain. This indicates that creating a truly convincing identity match remained difficult in 2016.
Factors Influencing Perception
- Gender Combinations: Intra-gender conversions (M-M, F-F) consistently outperformed cross-gender conversions in both metrics. Conversion to female targets was generally more successful, possibly due to acoustic characteristics or listener bias.
- Context Sensitivity: Naturalness ratings were found to be context-sensitive; a listener's score for one sample was often influenced by the quality of preceding samples in the session.
Conclusion and Future Work
The VCC 2016 results emphasize that voice conversion is not yet a "solved" problem. Achieving high naturalness and high similarity simultaneously remains a formidable task. Limitations of the study included a listener pool that was disproportionately female and British-English, which may have affected the perception of American-English stimuli. Future work suggests more exhaustive Multi-Dimensional Scaling (MDS) analysis on smaller system subsets and investigating the impact of listener demographics on voice perception.
Summary updated