Abstract
As Spoken Language Models (SLMs) evolve toward end-to-end architectures, maintaining stylistic consistency—such as specific emotions, accents, or speaking rates—across multi-turn interactions becomes critical for natural user experiences. This paper identifies and investigates "Style Amnesia," a phenomenon where SLMs initially follow stylistic instructions but rapidly revert to default behaviors over the course of a conversation. Through an automated evaluation framework, we demonstrate that while models retain the memory of instructions, they fail in execution. We further propose a "Recall Process" mitigation strategy that significantly reduces stylistic degradation.
Introduction
The transition from text-based Large Language Models (LLMs) to end-to-end Spoken Language Models (SLMs), such as GPT-4o and Gemini Live, has shifted expectations from purely verbal accuracy to paralinguistic consistency. Users expect AI assistants to maintain a professional tone, a specific regional accent, or a sense of urgency throughout an entire interaction. However, current research has largely focused on single-turn performance. This study identifies a significant gap: in back-and-forth dialogues, SLMs tend to drift back to a "neutral" default voice (typically a North American accent with neutral emotion) after only a few exchanges. We term this decay Style Amnesia.
The Style Amnesia Phenomenon
To quantify this effect, we evaluated five major SLMs (GPT-4o, GPT-4o mini, Gemini Live, Qwen2.5-Omni, and Step-Audio 2 mini) across various stylistic attributes, including emotion, accent, volume, and speed.
As shown in Figure 1, the Instruction-Following (IF) rate—the percentage of turns where the model successfully adheres to the requested style—drops precipitously as the conversation progresses. For many models, a style successfully adopted in the first turn is almost entirely lost by the fourth turn, regardless of the model's proprietary or open-source nature.
Methodology
We developed an automated evaluation framework to ensure consistent testing:
- User Simulator: A cascaded system (ASR + LLM + TTS) acted as the human interlocutor to provide standardized conversational prompts.
- Style Instructions: Models were instructed at the beginning of the session to maintain a specific attribute (e.g., "speak sadly" or "use an Indian accent").
- Automatic Judges: To avoid the cost and subjectivity of human-only evaluation, we utilized specialized models like Emotion2vec and Voxlect to verify stylistic adherence at every turn.
- Recall Probe: To distinguish between memory loss and execution failure, we inserted probes where models were asked to restate their assigned style before responding.
Key Results and Insights
Our analysis revealed several counter-intuitive findings regarding how SLMs process stylistic constraints.
Memory vs. Execution
One of the most striking insights is that models do not actually "forget" the instructions. When prompted in later turns to recall their assigned style, proprietary models showed near-perfect memory. However, their actual speech output failed to reflect that style. This suggests a disconnect between the model's linguistic understanding of a command and its paralinguistic execution in an end-to-end audio pipeline.
The System Message Paradox
In traditional text LLMs, system messages are the primary tool for enforcing persistent rules. Surprisingly, for SLMs, placing style instructions in the system message often led to worse performance than including them in the first user message. For instance, GPT-4o mini’s ability to maintain an Indian accent dropped by nearly 80% when the instruction was moved from the user prompt to the system prompt.
Attribute-Specific Degradation
Comprehensive evaluation of first-turn IF rates and degradation rates (D) across various stylistic attributes for different models. Source, p. 6
As illustrated in Figure 6, degradation is not uniform across all styles. Models exhibit a strong "default bias" toward neutral emotions and North American accents. Styles that deviate significantly from this center of gravity—such as "Sadness" or "Indian Accent"—suffer the most severe amnesia. For example, GPT-4o mini’s IF rate for sadness plummeted from 85% in the first turn to just 9% by the fourth.
Mitigation: The Recall Process
To bridge the gap between memory and execution, we tested a "Recall Process" where the model is forced to explicitly state its required style before generating a response. This "think-out-loud" approach significantly reduced degradation. For GPT-4o mini on the Indian accent task, the degradation rate was slashed from 49.7% to 14.9%, suggesting that explicit self-reminders can help the model maintain stylistic focus.
Conclusion
Style Amnesia represents a significant hurdle for the next generation of voice assistants. While current SLMs are capable of high-quality single-turn stylistic output, they struggle with long-term consistency. Our findings suggest that the "all-in-one" architecture of modern SLMs currently lacks the robust stylistic grounding found in cascaded systems. Future work must focus on embedding stylistic consistency directly into model architectures or training objectives to move beyond prompting-based mitigations.