Predicting character-appropriate voices for a TTS-based storyteller system

Erica Greene, Taniya Mishra, Patrick Haffner, Alistair ConkieInterspeech 2012

TL;DR

This paper tackles the challenge of automatically assigning distinct, character-appropriate voices in Text-to-Speech (TTS) storyteller systems for children's stories. They developed a data-driven approach using Amazon Mechanical Turk to identify salient character attributes and model their relationship with TTS voices, achieving significantly better-than-chance performance in predicting suitable voices.

What Problem Does This Paper Address?

Current TTS-based ebook readers often fall short of engaging human storytellers because they use a single, generic voice. To create truly immersive digital storytelling experiences, especially for children, systems need to go beyond basic narration. This involves expressing emotions, generating suspense, and crucially, assigning unique, character-appropriate voices to different characters within a story. This last aspect, automatically predicting the right voice for a character based on its attributes, is a largely unexplored but vital problem for enhancing comprehension and engagement in child-directed applications.

Methods

The authors propose a data-driven approach to predict character-appropriate TTS voices. Their method involves three key steps:

  1. Voice Set Composition: They analyzed common children's stories to identify recurring character archetypes and ensured their TTS voice collection adequately covered these, supplementing existing AT&T voices with others from public sources.
  2. Attribute Identification: To find the most salient attributes that listeners associate with specific voices, they used Amazon Mechanical Turk (MT). Participants listened to voice clips of characters from children's stories and described the voices using a mix of multiple-choice and free-response questions. They also augmented this with attributes from voice actor profiles. These were then clustered and refined into a set of representative adjectives.
  3. Attribute-to-Voice Modeling: They employed a Naive Bayes classifier to model the relationship between character attributes and TTS voices. This classifier takes a set of character attributes as input and outputs a ranked list of voices, from most to least appropriate. The authors chose Naive Bayes for its efficiency in ranking problems and its ability to handle situations where training data priors might not perfectly reflect testing conditions.

Datasets

  • Voice Actor Profiles: Used to extract descriptive adjectives for character attributes.
  • Children's Movies/TV Show Clips: Audio clips from these sources were used for the MT task to identify salient character attributes.
  • Project Gutenberg Children's Stories: Used to identify common character archetypes.
  • AT&T TTS Voices: The core set of synthetic voices used.
  • Publicly Available Voices: Supplemented the AT&T voices to improve coverage.
  • Newly Annotated Data: The authors created a dataset by having MT workers annotate their TTS voice collection with the identified salient attributes. This involved presenting audio clips of TTS voices speaking text and having workers rate them on various attribute scales.

Key Insights and Ideas

  • Salient Attributes are Key: The success of predicting character-appropriate voices hinges on identifying the right attributes that listeners intuitively associate with voices. The authors found that while obvious attributes like sex and age are important, others like "evilness," "silliness," and "gentleness" are crucial for distinguishing voices in children's stories.
  • Naive Bayes for Ranking: The Naive Bayes classifier proved to be a surprisingly effective and straightforward method for bridging the gap between attribute-based character descriptions and voice selection, particularly for ranking potential voices.
  • Text Content vs. Voice Quality: The authors took steps to control for the influence of text content on voice perception by using paired biased and neutral text excerpts. Their analysis showed that text content had minimal impact on attribute ratings, suggesting the focus on voice quality was successful.
  • Voice Set Coverage Matters: The initial analysis of common character archetypes highlighted the need for a diverse voice set, particularly for underrepresented categories like young, evil, and silly voices, to effectively serve a storytelling system.

Key Results

The evaluation showed that their Naive Bayes system performed significantly above chance. Specifically:

  • The system had over a 50% chance of returning the correct voice within its top 5 guesses.
  • The system had an 85% chance of returning the correct voice within its top 10 guesses.
  • The contribution of "sex and age" attributes alone accounted for about half of the improvement over random guessing.

Limitations and Future Work

  • Uniform Priors: The current model assumes a uniform prior distribution over voices, which might not reflect the true probability of a voice being suitable for a character in a large corpus of stories. The authors suggest exploring methods to determine more accurate priors.
  • Objective vs. Perceptual Evaluation: The current evaluation is objective. The authors plan to conduct perceptual studies with human subjects to measure the perceived appropriateness of the automatically selected voices, especially in multi-character exchanges.
  • Attribute Granularity: While the system uses salient attributes, characters in practice might have only a few distinguishing characteristics. The authors note that a heuristic of disregarding attributes with a neutral value (e.g., a "3" on a 1-5 scale) improved 1-best accuracy.
  • Future Perceptual Studies: The authors aim to use data from multiple-choice questions in their MT tasks to analyze acoustic prosodic characteristics of voices, further refining their understanding of how attributes relate to voice qualities like happiness and silliness.

Summary updated

BibTeX
@inproceedings{greene2012predicting,
  author = {Erica Greene and Taniya Mishra and Patrick Haffner and Alistair Conkie},
  title = {{Predicting character-appropriate voices for a TTS-based storyteller system}},
  booktitle = {Interspeech},
  year = {2012},
}

Loading related content...

Figure