StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models

Kazuki Yamauchi, Yusuke Ijima, Yuki Saito

Abstract

StyleCap is a novel end-to-end framework designed to generate natural language captions describing a speaker's style, such as pitch, pace, and volume. By bridging speech self-supervised learning (SSL) models with Large Language Models (LLMs), StyleCap moves beyond traditional emotion classification toward more nuanced and interpretable paralinguistic descriptions. Experimental results demonstrate that the framework significantly outperforms baseline models, particularly when utilizing WavLM features and sentence rephrasing augmentation.

Introduction

Human communication consists of two primary streams: linguistic (what is said) and paralinguistic (how it is said). While modern AI excels at transcribing words, describing the "how"—the subtle nuances of tone, tempo, and identity—remains a challenge. Traditional systems often treat this as a classification task, assigning reductive labels like "Happy" or "Sad."

This paper introduces automatic speaking-style captioning, a task requiring a model to generate descriptive sentences from audio clips. This "one-to-many" challenge is addressed by StyleCap, which translates complex audio features into natural language descriptions, providing a more flexible and reasoning-based approach to paralinguistic analysis.

Methodology

StyleCap is an architecture inspired by image-captioning frameworks like ClipCap. It consists of three integrated components:

  1. Speech Encoder: The model utilizes a pre-trained WavLM model to extract features from multiple SSL layers. These features are weighted and processed through an Aggregation Module (combining BLSTM and Multi-head Attention) to produce a fixed-length vector representing the speech style.
  2. Mapping Network: This component acts as a bridge, projecting the speech vector into the "prefix" space of an LLM. It essentially translates audio characteristics into a prompt format the language model can interpret.
  3. Text Decoder: The framework employs powerful LLMs, such as GPT-2 or Llama 2 (7B), to autoregressively generate the final caption based on the provided prefix.
The StyleCap architecture, illustrating the flow from input speech through the encoder, mapping network, and LLM-based text decoder.

To address the scarcity of diverse training data, the authors introduced Sentence Rephrasing Augmentation. Using Llama 2, existing human-annotated captions were rewritten into multiple variations, ensuring the model learns diverse linguistic representations for the same acoustic styles.

Key Results and Discussion

The "best" configuration of StyleCap—incorporating WavLM, Llama 2, and sentence rephrasing—consistently outperformed baseline Transformer-based encoder-decoder models across all standard NLP metrics.

  • Caption Accuracy: The model achieved a METEOR score of 0.479 and a CIDEr-D of 3.594, significantly higher than mel-spectrogram-based baselines.
  • Diversity and Robustness: Metrics for vocabulary diversity (distinct-1 and distinct-2) showed that sentence rephrasing augmentation was critical for generating varied and natural descriptions.
  • Zero-Label Learning: Interestingly, despite being trained only on natural language, the model implicitly learned to classify gender, pitch, speed, and volume with an average accuracy of 76.8%.
Quantitative results showing the performance of StyleCap variants compared to baselines using sentence rephrasing augmentation.

Conclusion

StyleCap demonstrates the power of combining speech SSL models with LLMs for paralinguistic tasks. By moving from classification to captioning, the framework provides a richer, more interpretable way to analyze speaking styles. Future work will focus on expanding the scope of styles to include complex emotional states and vocal markers of health, as well as developing more specialized evaluation metrics for style description accuracy.

Summary updated

BibTeX
@inproceedings{yamauchi2024stylecap,
  author = {Kazuki Yamauchi and Yusuke Ijima and Yuki Saito},
  title = {{StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=jgoin8YtpB},
}
Figure