Similar speaker selection technique based on distance metric learning with perceptual voice quality similarity

Yusuke Ijima, Mitsuaki Isogai, Hideyuki MizunoInterspeech 2012

TL;DR

This paper introduces a novel technique for selecting perceptually similar speakers by learning a transformation for acoustic features that aligns them with subjective voice quality ratings. This approach significantly reduces speaker selection errors compared to methods relying solely on acoustic similarity.

What Problem Does This Paper Address?

In speech synthesis, particularly for systems aiming to mimic a target speaker's voice, accurately identifying speakers with similar vocal characteristics is crucial. Traditional methods often rely on acoustic feature similarity (like MFCCs), but these don't always correlate with human perception of voice quality. This means two speakers might have acoustically similar feature distributions but sound quite different to a listener. The authors aim to bridge this gap by developing a speaker selection method that explicitly incorporates perceptual voice quality similarity.

Methods

The core of the proposed method is Distance Metric Learning (DML), specifically using Relevant Component Analysis (RCA). Here's a breakdown:

  • Utterance Vectors: They represent each speech segment using a GMM supervector. This is a powerful representation for text-independent speaker recognition, capturing statistical properties of the speech.
  • Perceptual Voice Quality Similarity: This is the "side information" for DML. They obtained a perceptual similarity matrix from a subjective experiment where listeners rated how similar pairs of speakers sounded.
  • Speaker Clustering: The perceptual similarity matrix is used to cluster speakers, defining "speaker classes."
  • RCA for Transformation: RCA is applied to learn a global linear transformation matrix. The goal of RCA is to minimize the distance between vectors within the same speaker class while maximizing the distance between different classes. In this context, "classes" are defined by the perceptual voice quality similarity clusters.
  • Feature Space Transformation: The learned RCA matrix transforms the original acoustic feature space into a new space where perceptual similarity is better represented.
  • Speaker Selection: After transforming the input speech's utterance vector using the learned matrix, speaker selection is performed using Euclidean distance in this transformed space, typically with a k-Nearest Neighbors (kNN) classifier.

Datasets

  • NTT-AT Japanese multi-speaker's speech database: This database contains speech data from 62 female speakers.
  • Training Data: 30 sentences from 61 speakers.
  • Evaluation Data: 30 sentences from the remaining speaker (not in training), used in a leave-one-out cross-validation setup.
  • Subjective Evaluation Data: Used to create the perceptual voice quality similarity matrix. The paper mentions this was a 3-point scale (3=very similar, 2=slightly similar, 1=very dissimilar) and that each speaker had at least one perceptually similar speaker.

Key Insights and Ideas

  • Acoustic vs. Perceptual Similarity: The fundamental insight is that acoustic feature similarity (e.g., MFCCs) is a poor proxy for perceptual voice quality similarity. This necessitates a method that explicitly learns this relationship.
  • DML as a Bridge: Distance Metric Learning, particularly RCA in this case, is shown to be an effective tool for learning a transformation that maps acoustic features to a space where perceptual similarity is more accurately reflected.
  • GMM Supervectors are Powerful: The GMM supervector proved to be a better utterance representation for speaker selection than a simpler "statistical vector" (mean, variance, quartile), highlighting its ability to capture speaker characteristics.
  • The Importance of "Side Information": The success of DML hinges on having relevant side information. Here, the subjective perceptual similarity ratings served as this crucial information.

Key Results

  • Error Reduction: The proposed method, using DML with RCA, achieved a significant reduction in speaker selection error rates. Specifically, the error rate (defined as selecting a speaker with low similarity, < 2) was reduced from 23.33% to 9.41%, resulting in a 59.6% error reduction rate.
  • Improved Average Similarity: The average similarity between the input speaker and the selected speaker was improved by using the DML-based transformation.
  • Feature Combination: Combining acoustic features like cepstral coefficients (cep), cepstral coefficients from low frequencies (cep4k), and the ratio of low-frequency power (BSP) generally improved performance, especially when used with the DML approach.

Limitations and Future Work

  • Global Transformation: The paper notes that RCA trains a global transform matrix. This might not capture subtle differences in how acoustic features relate to perceptual similarity across different speaker classes. The authors suggest that future work could explore DML techniques that can train transforms considering these class-specific differences.
  • Future DML Techniques: The authors plan to investigate other DML techniques beyond RCA to further improve speaker selection performance.
  • Arbitrary Speaker Synthesis: A key future application is to leverage this improved perceptual similar speaker selection for more accurate arbitrary speaker speech synthesis.

Summary updated

BibTeX
@inproceedings{ijima2012similar,
  author = {Yusuke Ijima and Mitsuaki Isogai and Hideyuki Mizuno},
  title = {{Similar speaker selection technique based on distance metric learning with perceptual voice quality similarity}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure