VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning

Xudong Tang, Matthew GroharXiv 2026 · arXiv

TL;DR

The authors introduce VIPBench, a large-scale benchmark of 124,876 human judgments designed to measure how people—not just metadata labels—perceive voice identity. Their findings reveal a significant gap between "ground truth" speaker labels and human perception, particularly for AI voice clones, which listeners perceived as a different person 41% of the time.

What Problem Does This Paper Address?

Traditionally, speaker recognition systems are trained and evaluated on metadata labels: if a recording was produced by Person A, the "correct" answer is Person A. However, voice identity is a psychological construct as much as a physical one. Factors like fundamental frequency shifts or emotional variability can make a single speaker sound like two different people.

In the age of high-fidelity voice cloning, this gap becomes a security and legal liability. A bank’s voice authentication system might accept a synthetic clone because it matches the metadata-trained model, even if a human listener would immediately flag it as sounding "off." Existing benchmarks like VoxCeleb lack human perceptual data, leaving researchers in the dark about how well their models actually align with human experience.

Methods

The researchers developed a rigorous protocol to evaluate 11 publicly available speech models (including supervised, self-supervised, and weakly supervised architectures).

They defined four evaluation tasks:

  1. Continuous $P(\text{same})$ prediction: How well does a model's cosine similarity score correlate with the proportion of humans who think a pair is the same person?
  2. Platt-calibrated verification: Can the model's similarity scores be mapped to a probability that accurately reflects human majority opinion?
  3. Speaker-level representational similarity: Do the pairs of speakers that humans find similar also sit close together in the model's embedding space?
  4. Real-to-synthetic calibration transfer: Does a model’s relationship to human judgment hold steady when moving from real recordings to synthetic clones and morphs?

Datasets

The core contribution is VIPBench, which differs from previous benchmarks by focusing on human perception at scale.

Table 1: The table compares VIPBench with related benchmarks. The speaker-verification and deep- fake benchmarks have metadata labels and no human judgments. CDPAM has human judgments of audio quality, and Fake2M and THINGS-data are image benchmarks with human judgments from an adjacent field. The Voice Conversion Challenges (VCC) and VoiceMorph also have human identity judgments, with fewer judgments and listeners than VIPBench, and only VoiceMorph has morphs. VIPBench is the first voice benchmark with human identity judgments on real recordings, AI voice clones, and voice morphs at a scale of more than 100,000 judgments. The columns are the size of the source corpus (# Speakers, # Stimuli), the human annotations (Human judgments, # Participants), the stimulus content (AI-gen, Morphs), and the evaluation Target. A check mark means that the benchmark has what the column names, and a cross means that it does not. A dash means that the entry does not apply or was not publicly reported. The citations are in Sections 1 and 2.
  • Scale: 124,876 judgments from 1,290 listeners.
  • Stimuli: 9,800 voice pairs from 100 public figures, stratified by gender, age, and sociophonetic groups (e.g., African American English, Southern American English).
  • Content Types: Real recordings, AI voice clones (generated via Cartesia Sonic 3), and "morphs" (weighted mixtures of two different speakers).
  • Comparison: Unlike VoxCeleb (metadata only) or ASVspoof (real vs. synthetic labels), VIPBench provides a "perceived sameness" target.

Key Insights and Ideas

  • The Clone Divergence: The most striking finding is that humans and metadata labels disagree most on AI clones. In 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people.
  • The EER Paradox: Models that perform best on traditional metadata benchmarks (low Equal Error Rate) don't necessarily align best with humans. For example, the resemblyzer model has a relatively high 12% error rate on metadata but aligns with human perception as well as state-of-the-art models with error rates below 2%.
  • Supervised vs. SSL: Supervised models (trained explicitly to distinguish speakers) align much better with human perception than self-supervised learning (SSL) models like HuBERT or wav2vec 2.0.
  • Layer Sensitivity: For SSL models, the best alignment with human identity perception occurs in the earliest layers, suggesting that higher-level semantic features in later layers might actually discard identity-relevant acoustic information.

Key Results

The researchers evaluated 11 models across their four tasks. As shown in the table below, resemblyzer emerged as the top performer for human alignment.

Table 4: Performance of eleven models across Tasks 1-3.
  • Noise Ceiling: No model reached the "noise ceiling" (the limit of human reliability). The best model explained only 61% of the explainable variance ($R^2/\rho_{SB}$).
  • Human Agreement: The best model (resemblyzer) agreed with the human majority on 79% of pairs, outperforming 72% of individual human listeners.
  • Calibration Issues: Most models' raw cosine similarities are much higher than human $P(\text{same})$ scores, requiring significant calibration (Platt scaling) to match human expectations.

Limitations and Future Work

The authors acknowledge several limitations:

  • Demographics: The dataset is limited to American English speakers and public figures.
  • System Specificity: The main main set uses a single TTS system (Cartesia); while a small extension used CosyVoice 2, the results for Task 4 (transfer) might be system-dependent.
  • Training Targets: Currently, no models are trained directly on human judgments. The authors suggest that doing so could close the gap toward the noise ceiling.

This work serves as a call to action for the speech community: to build truly secure and human-aligned voice systems, we must move beyond metadata and start accounting for how voices are actually heard.

Summary updated

Abstract

Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two voices as the same person has consequences for intellectual property rights, cybersecurity, and everyday life. We introduce the Voice Identity Perception benchmark (VIPBench), which contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs from 100 public figures. The pairs include real recordings, voice clones generated by a text-to-speech system, and morphs between two speakers' voices. We also define four tasks that evaluate the alignment between speaker-representation models and human identity perception. The first two tasks measure how well a model's similarity score predicts the proportion of listeners who respond "same" and the majority response. The other two measure whether a model and listeners find the same speaker pairs similar, and whether a model's relationship to listener judgments holds equally for real and synthetic recordings. We evaluate eleven publicly available speech models. Listener judgments diverge from speaker labels most often for voice clones: in 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people. No model reaches the noise ceiling; the best model explains 61% of the explainable variance. Models trained with speaker labels align with listeners better than self-supervised models do, yet a model with a 12% error rate on speaker labels aligns with listeners as well as models with error rates below 2%. VIPBench enables the evaluation of speaker models against human perception on real, cloned, and morphed speech, and our results motivate training speaker models directly on listener judgments.
BibTeX
@misc{tang2026vipbench,
  author = {Xudong Tang and Matthew Groh},
  title = {{VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning}},
  year = {2026},
  eprint = {2611.05638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05638},
}
Figure