STYLECAP: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-Supervised Learning Models

Kazuki Yamauchi, Yusuke Ijima, Yuki Saito

TL;DR

StyleCap is a new end-to-end framework that generates natural language captions describing a speaker's style (e.g., "a high-pitched male voice speaking very quickly"). By bridging speech self-supervised learning (SSL) models with Large Language Models (LLMs), it moves beyond simple emotion classification toward more nuanced, interpretable descriptions of paralinguistic information.

What Problem Does This Paper Address?

When we listen to someone, we process two streams of information: what they are saying (linguistic) and how they are saying it (paralinguistic). While AI has become excellent at transcribing words, it struggles to describe the "how"—the pitch, pace, volume, and tone that convey emotion and identity.

Existing systems usually treat this as a classification problem, slapping a label like "Happy" or "Sad" on a clip. However, these labels are often too reductive and lack "reasoning." This paper introduces automatic speaking-style captioning, a task that requires a model to listen to an audio clip and generate a descriptive sentence. This is a "one-to-many" challenge: a single speaking style can be described in dozens of different ways, and until now, there hasn't been a robust end-to-end architecture to handle this speech-to-text translation.

Methods

The authors propose StyleCap, an architecture inspired by ClipCap (a popular image-captioning model). It consists of three main components:

  1. Speech Encoder: Instead of using raw audio, the model uses a pre-trained WavLM model. It extracts features from various layers of the SSL model, weights them, and passes them through an "Aggregation Module" (a mix of BLSTM and Multi-head Attention) to create a single, fixed-length vector representing the speech style.
  2. Mapping Network: This component acts as a bridge. It takes the speech vector and projects it into the "prefix" space of an LLM. Essentially, it translates audio features into a "visual prompt" that the language model can understand.
  3. Text Decoder: The authors experimented with GPT-2 and Llama 2 (7B). The LLM receives the prefix from the mapping network and autoregressively generates the caption.

To handle the lack of diverse training data, they introduced Sentence Rephrasing Augmentation. They used Llama 2 to take existing human-annotated captions and rewrite them in multiple ways, ensuring the model learns that "speaking fast" and "at a high tempo" mean the same thing.

Datasets

  • PromptSpeech (Real Version): This is the primary dataset, containing 26,588 human-annotated style prompts.
  • LibriTTS: The underlying multi-speaker speech corpus from which the audio samples in PromptSpeech are derived.
  • Llama 2-Chat (7B): Used specifically for the data augmentation (rephrasing) process.

Key Insights and Ideas

  • SSL Features are Essential: The researchers found that WavLM significantly outperformed traditional mel-spectrograms and x-vectors (speaker embeddings). WavLM’s pre-training on massive amounts of speech allows it to capture subtle paralinguistic nuances that simpler features miss.
  • LLM "Richness" Matters: Moving from the relatively small GPT-2 to the 7-billion parameter Llama 2 provided a substantial boost in performance. Larger LLMs aren't just better at grammar; they are better at "understanding" the conditional prefix provided by the speech encoder.
  • The Power of Rephrasing: Data augmentation wasn't just a minor tweak. By rephrasing the training captions, the model became much better at generating diverse and accurate descriptions, effectively tackling the "one-to-many" mapping problem inherent in natural language.
  • Style Factor Sensitivity: The authors noted that if the model misclassifies even one factor (like getting the volume wrong while getting pitch and speed right), the overall quality of the caption (measured by METEOR scores) drops sharply.

Key Results

The "best" version of StyleCap (using WavLM, Llama 2, and data augmentation) outperformed the baseline Transformer-based encoder-decoder models across all metrics:

  • Caption Accuracy: Achieved a METEOR score of 0.479 and a CIDEr-D of 3.594, significantly higher than the mel-spectrogram baselines.
  • Diversity: The "distinct-1" and "distinct-2" metrics showed that the model generated a wider variety of vocabulary and phrasing compared to models trained without sentence rephrasing.
  • Zero-Label Learning: Even though the model was only trained on natural language descriptions, it "accidentally" learned to classify gender, pitch, speed, and volume with high accuracy (averaging 76.8% across factors).

Limitations and Future Work

  • Evaluation Metrics: The authors admit that standard NLP metrics like BLEU or ROUGE might not be the best way to judge if a style description is "correct" in a human sense. They suggest exploring LLM-based evaluation (like G-Eval) in the future.
  • Scope of Styles: The current work focuses on physical attributes (pitch, speed, volume). Future iterations could expand to more complex paralinguistic info, such as specific emotional states or even vocal markers of health conditions.
  • Dataset Scarcity: While PromptSpeech is a great start, the field needs much larger-scale speech-caption paired datasets to reach the level of performance seen in image captioning.

Summary updated

Abstract

We propose StyleCap, a method to generate natural language descriptions of speaking styles appearing in speech. Although most of conventional techniques for para-/non-linguistic information recognition focus on the category classification or the intensity estimation of pre-defined labels, they cannot provide the reasoning of the recognition result in an interpretable manner. StyleCap is a first step towards an end-to-end method for generating speaking-style prompts from speech, i.e., automatic speaking-style captioning. StyleCap is trained with paired data of speech and natural language descriptions. We train neural networks that convert a speech representation vector into prefix vectors that are fed into a large language model (LLM)-based text decoder. We explore an appropriate text decoder and speech feature representation suitable for this new task. The experimental results demonstrate that our StyleCap leveraging richer LLMs for the text decoder, speech self-supervised learning (SSL) features, and sentence rephrasing augmentation improves the accuracy and diversity of generated speaking-style captions. Samples of speaking-style captions generated by our StyleCap are publicly available 1.
BibTeX
@inproceedings{yamauchi2024stylecap,
  author = {Kazuki Yamauchi and Yusuke Ijima and Yuki Saito},
  title = {{STYLECAP: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-Supervised Learning Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=jgoin8YtpB},
}

Loading related content...

Figure