A Comparative Study of Statistical Conversion of Face to Voice Based on Their Subjective Impressions

Yasuhito Ohsugi, Daisuke Saito, Nobuaki MinematsuInterspeech 2018

TL;DR

This paper explores how to generate a voice that subjectively matches a given face, moving beyond purely objective mappings. They investigate different ways to represent faces and voices, and compare statistical models for converting between them, finding that subjective impressions are key and that the optimal model complexity depends on the data.

What Problem Does This Paper Address?

As voice-based user interfaces (VUIs) become more sophisticated, there's a growing desire to make them more engaging and human-like. One way to achieve this is by personifying them with visual features, like a face. However, simply attaching any face to any voice doesn't create a natural interaction. The core question this paper tackles is: what kind of face is perceived as a good match for a particular voice quality, and conversely, what voice quality is perceived as a good match for a given face? This is particularly relevant for applications like embodied conversational agents (ECAs) or even talking toys. The authors focus on subjective impressions – what humans perceive as a good match – rather than purely objective acoustic or visual features.

Methods

The researchers propose and compare several methods for converting face features to voice features.

  • Face Feature Extraction:
    • Iconified Face Features (IFFs): This method uses facial landmarks detected by a Constrained Local Neural Field (CLNF) model. These landmarks are then converted into "iconified facial parts" (like eyes, nose, mouth) represented by 15 geometrical parameters. These parameters are then compressed using Principal Component Analysis (PCA).
  • CVAE Features: A Conditional Variational Autoencoder (CVAE) is used to embed a face image into a latent feature vector. The mean vector of the posterior distribution from the CVAE is used as the face feature.
  • Voice Feature Extraction:
    • Eigenvoice: This is a well-established method for representing speaker characteristics. It's calculated using PCA on supervectors of speaker-dependent Gaussian Mixture Models (SD-GMMs). The idea is to convert face features into these "eigenvoice" representations.
  • Conversion Models:
    • Gaussian Mixture Model (GMM): A standard approach for modeling joint distributions between two feature spaces.
  • Probabilistic Canonical Correlation Analysis (pCCA): A method that models the latent relationship between two sets of variables.
  • Mixture of Probabilistic Canonical Correlation Analysis (mPCCA): An extension of pCCA that uses multiple mixtures to capture more complex relationships.

The conversion process is framed as "face to eigenvoice conversion," and the models are trained on a parallel corpus of subjective mappings from faces to voices.

Datasets

  • Face Images: The MORPH database (54,147 images) was used for training face feature extractors. For the main experiments, 71 face images from the Morishima laboratory at Waseda University were used.
  • Voice Data: The JNAS database (73 speakers) was used for voice features.
  • Subjective Mappings: A novel dataset was created by collecting subjective mappings from 17 Japanese adult subjects. These subjects were asked to select the voice they felt best matched a given face image. This dataset consists of 64 paired data points for training and 7 for testing.

Key Insights and Ideas

  • Subjectivity is Paramount: The study strongly suggests that subjective impressions are crucial for face-to-voice matching. The collected data revealed that subjects often mapped multiple faces to a single preferred voice, indicating a "many-to-one" relationship, and this preference was highly individual.
  • Feature Representation Matters: The choice of face feature representation impacts conversion performance. CVAE features with a lower dimensionality (dµ=3) showed slightly better results than IFFs.
  • Model Complexity vs. Data Size: The performance of GMM and mPCCA models was influenced by the size of the subjective mapping data. With limited data, simpler models might perform better, while more complex models like mPCCA might require more data to effectively capture intricate relationships.
  • Domain-Specific Mixture Numbers: For mPCCA, the optimal number of mixtures (components) might differ between the face feature space and the voice feature space. This suggests that a more flexible approach to model complexity per domain could be beneficial.

Key Results

  • Subjective Mapping Findings: The subjective mapping experiment revealed that each subject had a preferred voice, and different faces were mapped to this single voice. This supports prior findings on the variability of identity matching.
  • Conversion Performance:
    • CVAE-µ with a dimensionality of 3 (dµ=3) performed slightly better than PC-IFF features.
  • CVAE-µ with a dimensionality of 10 (dµ=10) performed worse than both PC-IFF and CVAE-µ (dµ=3).
  • When comparing conversion models for CVAE-µ (dµ=10), pCCA outperformed GMM and mPCCA.
  • For PC-IFF and CVAE-µ (dµ=3), GMM was slightly better than pCCA, and both were better than mPCCA. This is attributed to the limited size of the subjective mapping dataset.
  • Qualitative Findings: The paper demonstrates that manipulating face features (like PC-IFF or CVAE-µ) can generate varied facial appearances, and similarly, manipulating eigenvoices can produce different vocal qualities.

Limitations and Future Work

  • Data Size: The authors acknowledge that the size of the collected subjective mapping data was not sufficient to effectively train more complex models like GMM and mPCCA, especially for higher-dimensional face features.
  • Feature Importance: While IFF features allow for interpretation of individual dimensions, the paper suggests future work to examine which specific face features play the most important roles in face-to-voice conversion.
  • Optimal Mixture Numbers: The paper highlights the need to investigate domain- or feature-dependent numbers of mixtures for mPCCA to better characterize the relationships between face and voice spaces.
  • Larger Datasets: Future work will involve collecting a larger number of manual face-to-voice mappings to better understand the impact of data size on conversion performance.

Summary updated

BibTeX
@inproceedings{ohsugi2018comparative,
  author = {Yasuhito Ohsugi and Daisuke Saito and Nobuaki Minematsu},
  title = {{A Comparative Study of Statistical Conversion of Face to Voice Based on Their Subjective Impressions}},
  booktitle = {Interspeech},
  year = {2018},
}

Loading related content...

Figure