Abstract
In human communication, paralinguistic features such as emotional tone, speed, and volume are essential for conveying meaning beyond literal text. However, current Large Language Models (LLMs) remain largely "style-blind," treating identical transcripts the same regardless of how they were spoken. This paper introduces Spoken-LLM, a framework that enables LLMs to perceive and respond to varied speaking styles. Central to this work is StyleTalk, a novel dataset featuring identical sentences paired with different speaking styles and corresponding responses. Our results demonstrate that Spoken-LLM effectively adjusts its output based on the speaker's delivery, significantly outperforming text-only baselines in stylistic accuracy and human preference.
Introduction
In a typical conversation, the phrase "It looks like it might rain" can elicit vastly different reactions depending on the speaker's tone. A cheerful delivery might prompt a hopeful response about the weather clearing, while a sad or frustrated tone might trigger a sympathetic acknowledgement of ruined plans. While LLMs have mastered textual semantics, they often fail to capture these nuances because they rely on speech-to-text transcriptions that strip away emotional subtext.
To bridge this gap, researchers from National Taiwan University developed Spoken-LLM. This framework moves beyond "content-centric" tasks like simple transcription toward "style-centric" interaction, allowing models to "hear" the emotional and rhythmic qualities of human speech.
Methodology
Architecture
The Spoken-LLM architecture integrates a pre-trained LLM with a specialized speech encoder. The researchers utilized Llama 2-Chat (7B) as the primary reasoning engine and emotion2vec as the acoustic feature extractor. To align these two modalities, a lightweight Connector module projects speech embeddings into the LLM’s input space.
To capture fine-grained temporal nuances, the model employs chunk-level embeddings. Rather than averaging an entire audio clip into a single vector, the system processes snippets of audio, allowing the LLM to perceive the timing of emotional cues and shifts in pace.
Two-Stage Training
The model undergoes a specialized training regimen to ensure it understands the relationship between sound and meaning:
- Stage 1 (Style Alignment): The model is trained to identify the style (emotion, speed, volume) of the input speech. This stage teaches the LLM to interpret what the speech embeddings represent.
- Stage 2 (Spoken Response Modeling): The model learns to predict both the text of the response and the appropriate style in which that response should be delivered.
To maintain efficiency, the researchers used LoRA (Low-Rank Adaptation), fine-tuning only a small fraction of the LLM's weights while keeping the core parameters frozen.
The StyleTalk Dataset
A primary contribution of this research is the StyleTalk dataset, designed specifically for speech-to-speech conversational generation. Unlike existing datasets that focus on emotion recognition (e.g., IEMOCAP), StyleTalk focuses on how style dictates the logical flow of dialogue.
- The "Same-Text, Different-Style" Feature: The dataset contains scenarios where the same dialogue context and input sentence lead to different responses based on the input style.
- Data Pipeline: The authors used GPT-4 to generate dialogue scripts with style annotations, synthesized the audio using Microsoft Azure’s expressive TTS, and employed human annotators via Amazon Mechanical Turk to filter for naturalness. The final curated set includes approximately 2,000 high-quality samples, supported by a larger "unfiltered" set of 16,000 samples used for model warmup.
Key Results and Insights
Performance Metrics
Spoken-LLM demonstrated a clear superior ability to handle stylistic nuances compared to traditional models:
- Style Accuracy: The model achieved an F1 score of 49.6 in predicting the correct response emotion, a massive improvement over the text-only baseline's score of 17.5.
- Linguistic Quality: The "chunk-based" version of the model achieved a 4.0 BLEU score, matching the performance of a theoretical "upper bound" cascaded model that was given ground-truth style labels manually.
- Response Diversity: The model showed a significantly lower "self-BLEU" score (10.9) compared to text-only models (100.0), proving it can generate diverse responses for the same text when the input style changes.
Human Evaluation
In A/B testing, human listeners consistently preferred Spoken-LLM’s responses over text-only alternatives. Annotators noted that the model's responses felt more "reasonable and proper," successfully mirroring or complementing the emotional state of the speaker.
Discussion and Limitations
The research highlights that speaking style is not merely "flavor" but a functional part of language semantics. The study found that certain styles, like "cheerful," lead to a wider variety of possible responses, while "sad" styles often elicit more predictable, empathetic "bottleneck" responses.
Despite these advances, several limitations remain. The current dataset relies on synthetic speech, which lacks the "messiness" (stutters, filler words, background noise) of real-world human interaction. Additionally, the model follows a strict turn-based format, missing the backchanneling and interruptions common in natural conversation. Future work will explore direct audio token generation to capture even more subtle vocal nuances without relying on intermediate style tags.
Conclusion
By integrating acoustic style perception directly into the LLM's reasoning process, Spoken-LLM represents a significant step toward more natural and emotionally intelligent AI assistants. The StyleTalk dataset provides a vital foundation for future research into how machines can better understand the complexities of human vocal expression.
Summary updated