VoxSim: A perceptual voice similarity dataset

Junseok Ahn, Youkyum Kim, Yeunju Choi, Doyeop Kwak, Ji‐Hoon Kim, Seongkyu Mun, Joon Son Chung

Abstract

This paper introduces VoxSim, a dataset of perceptual voice similarity ratings. Recent efforts to automate the assessment of speech synthesis technologies have primarily focused on predicting mean opinion score of naturalness, leaving speaker voice similarity relatively unexplored due to a lack of extensive training data. To address this, we generate about 41k utterance pairs from the VoxCeleb dataset, a widely utilised speech dataset for speaker recognition, and collect nearly 70k speaker similarity scores through a listening test. VoxSim offers a valuable resource for the development and benchmarking of speaker similarity prediction models. We provide baseline results of speaker similarity prediction models on the VoxSim test set and further demonstrate that the model trained on our dataset generalises to the out-of-domain VCC2018 dataset.

Introduction

In the development of speech synthesis (TTS) and voice conversion systems, researchers strive for two primary qualities: naturalness and speaker similarity. While metrics for naturalness are well-established, measuring how much a synthesized voice resembles a target speaker remains challenging.

Currently, speaker verification models like ECAPA-TDNN are often used to calculate cosine similarity between voice embeddings. However, there is a significant discrepancy between AI and human perception. Modern AI models achieve an Error Rate (EER) of less than 1%, whereas humans typically range between 17% and 26%. AI models are often "too good," detecting acoustic nuances that humans ignore. Consequently, objective scores from these models frequently fail to align with human judgment. Existing datasets for training perceptual similarity models are limited in scale, often featuring fewer than 15 speakers.

Methods and Data Collection

To address the need for large-scale perceptual data, we created VoxSim. The dataset construction involved several key steps:

  1. Pair Generation: We sampled 41,578 utterance pairs from the VoxCeleb1 dataset. To ensure diversity, we included both "same speaker" and "different speaker" pairs, with a 1.5x bias toward the latter.
  2. Human Labeling: Twelve evaluators rated the pairs on a 6-point Likert scale. The scale ranges from "Definitely different" (1) to "Definitely the same" (6).
  3. Quality Control: Evaluator integrity was monitored by tracking their speaker verification error rates. If an evaluator's performance drifted significantly from the average, they were required to re-assess their samples.
The speaker annotation interface used for collecting similarity ratings, featuring a 6-point Likert scale.

Dataset Analysis

VoxSim contains 69,409 ratings for 41,578 pairs across 1,251 speakers. A key feature of the dataset is the inclusion of "unseen" speakers in the test set to evaluate model generalization.

Distribution of human similarity scores across individual ratings, averages, and speaker pair types.

The distribution of scores (shown above) illustrates the "soft boundary" effect in human perception. Unlike the sharp clusters formed by standard speaker recognition models, human ratings reflect a continuum of similarity. This suggests that while an AI might definitively distinguish two speakers, humans may still perceive them as somewhat similar.

Modeling and Results

We benchmarked three architectures—ECAPA-TDNN, WavLM-ECAPA, and SVSNet—by fine-tuning them to predict human similarity scores.

  • Top Performance: The WavLM-ECAPA model, which leverages self-supervised learning (SSL) features, achieved the best results with a Linear Correlation Coefficient (LCC) of 0.835 on the VoxSim test set.
  • Impact of Fine-tuning: Fine-tuning on VoxSim improved the LCC of the ECAPA-TDNN model from 0.768 to 0.829, demonstrating the value of the perceptual labels.
  • Generalization: Despite being trained on natural speech, models trained on VoxSim generalized well to the synthetic speech in the VCC2018 dataset. The VoxSim-trained ECAPA-TDNN reached an LCC of 0.605, outperforming the specialized SVSNet (0.574).

Conclusion

VoxSim provides a substantial resource for aligning automated voice evaluation with human perception. Our results indicate that pre-training on speaker recognition followed by fine-tuning on perceptual similarity is a robust approach for this task. Future work will focus on incorporating synthetic speech artifacts to further refine these models for modern TTS evaluation.

Summary updated

Abstract

This paper introduces VoxSim, a dataset of perceptual voice similarity ratings. Recent efforts to automate the assessment of speech synthesis technologies have primarily focused on predicting mean opinion score of naturalness, leaving speaker voice similarity relatively unexplored due to a lack of extensive training data. To address this, we generate about 41k utterance pairs from the VoxCeleb dataset, a widely utilised speech dataset for speaker recognition, and collect nearly 70k speaker similarity scores through a listening test. VoxSim offers a valuable resource for the development and benchmarking of speaker similarity prediction models. We provide baseline results of speaker similarity prediction models on the VoxSim test set and further demonstrate that the model trained on our dataset generalises to the out-of-domain VCC2018 dataset.
BibTeX
@inproceedings{ahn2024voxsim,
  author = {Junseok Ahn and Youkyum Kim and Yeunju Choi and Doyeop Kwak and Ji‐Hoon Kim and Seongkyu Mun and Joon Son Chung},
  title = {{VoxSim: A perceptual voice similarity dataset}},
  booktitle = {Interspeech},
  year = {2024},
  doi = {10.21437/interspeech.2024-646},
}
Figure