SVSNet: An End-to-End Speaker Voice Similarity Assessment Model

Cheng-Hung Hu, Yuhuai Peng, Junichi Yamagishi, Yu Tsao, Hsin‐Min Wang

Abstract

Neural evaluation metrics derived for numerous speech generation tasks have recently attracted great attention. In this paper, we propose SVSNet, the first end-to-end neural network model to assess the speaker voice similarity between converted speech and natural speech for voice conversion tasks. Unlike most neural evaluation metrics that use hand-crafted features, SVSNet directly takes the raw waveform as input to more completely utilize speech information for prediction. SVSNet consists of encoder, co-attention, distance calculation, and prediction modules and is trained in an end-to-end manner. The experimental results on the Voice Conversion Challenge 2018 and 2020 (VCC2018 and VCC2020) datasets show that SVSNet outperforms well-known baseline systems in the assessment of speaker similarity at the utterance and system levels.

Introduction

In Voice Conversion (VC), the objective is to transform a source speaker's audio to match a target speaker's identity while preserving the linguistic content. Evaluating the success of this transformation is a significant challenge. Traditional objective metrics, such as Mel-cepstral distance, often fail to align with human perception of "voice similarity." Consequently, researchers rely on expensive and time-consuming subjective listening tests.

While models like MOSNet have successfully automated "naturalness" prediction, a dedicated end-to-end model for speaker similarity has been missing. SVSNet addresses this gap by overcoming two major hurdles: the loss of phase information in spectrogram-based features and the difficulty of comparing asymmetric utterances with varying lengths or linguistic content.

Proposed Architecture

SVSNet employs an end-to-end architecture designed to process raw audio and maintain symmetry in comparison.

The SVSNet architecture (a), including the encoder (b) and residual-skipped-WaveNet (rSWC) modules (c).

The model consists of four primary stages:

  1. Encoder (Raw Waveform Processing): Utilizing SincNet layers, the model learns band-pass filters directly from the raw signal, preserving crucial phase information. This is followed by residual-skipped-WaveNet (rSWC) blocks and a BLSTM to capture both local acoustic nuances and long-term temporal dependencies.
  2. Co-Attention Module: A key innovation of SVSNet is the bidirectional co-attention mechanism. It aligns the test and reference utterances in both directions, ensuring the model can handle content mismatches and remains symmetric (i.e., the similarity score remains the same even if the input order of the two clips is swapped).
  3. Distance Calculation: The model calculates the 1-norm distance between the mean representations of the aligned sequences.
  4. Prediction: The final module outputs a similarity score. The authors evaluated both regression-based (SVSNet-R) and classification-based (SVSNet-C) variants.

Experimental Results

The model was rigorously tested using data from the Voice Conversion Challenges (VCC2018 and VCC2020). VCC2018 provided over 21,000 pairs for training, while VCC2020 served to test generalization across different languages and neural vocoders.

Performance comparison of SVSNet and MOSNet at utterance and system levels.

As shown in the results table, SVSNet significantly outperforms the MOSNet baseline. On the VCC2018 dataset, SVSNet(R) achieved a system-level Linear Correlation Coefficient (LCC) of 0.965 and a Spearman’s Rank Correlation Coefficient (SRCC) of 0.903. These high correlation values indicate that SVSNet’s rankings of VC systems are remarkably close to those of human panels.

Discussion and Future Work

Ablation studies confirmed that raw audio input via SincNet provides superior performance compared to traditional spectrograms, suggesting that "hand-crafted" features discard valuable speaker-specific information. Furthermore, the co-attention mechanism proved essential for handling utterances with different text content.

While SVSNet shows strong generalization, performance can vary when encountering synthesis technologies (vocoders) not present in the training data. Future work includes integrating psychoacoustic theories to improve robustness and utilizing SVSNet as a loss function to directly optimize VC models for speaker similarity.

Summary updated

Abstract

Neural evaluation metrics derived for numerous speech generation tasks have recently attracted great attention. In this paper, we propose SVSNet, the first end-to-end neural network model to assess the speaker voice similarity between converted speech and natural speech for voice conversion tasks. Unlike most neural evaluation metrics that use hand-crafted features, SVSNet directly takes the raw waveform as input to more completely utilize speech information for prediction. SVSNet consists of encoder, co-attention, distance calculation, and prediction modules and is trained in an end-to-end manner. The experimental results on the Voice Conversion Challenge 2018 and 2020 (VCC2018 and VCC2020) datasets show that SVSNet outperforms well-known baseline systems in the assessment of speaker similarity at the utterance and system levels.
BibTeX
@article{hu2022svsnet,
  author = {Cheng-Hung Hu and Yuhuai Peng and Junichi Yamagishi and Yu Tsao and Hsin‐Min Wang},
  title = {{SVSNet: An End-to-End Speaker Voice Similarity Assessment Model}},
  journal = {IEEE Signal Processing Letters},
  year = {2022},
  doi = {10.1109/lsp.2022.3152672},
}
Figure