SAGE: Steering and Refining Dialog Generation with State-Action Augmentation

Yizhe Zhang, Navdeep JaitlyarXiv 2025 · DOI · arXiv

Abstract

SAGE (State Augmented GEneration) is a framework designed to enhance the social intelligence and strategic planning of conversational agents. By introducing a "State-Action Chain" (SAC), the model predicts latent variables—user emotional states and intended bot strategies—prior to response generation. This approach results in dialogue that is more concise, empathetic, and human-like compared to standard instruction-tuned Large Language Models (LLMs).

Introduction

While Large Language Models (LLMs) excel at task-oriented goals like coding or mathematical reasoning, they often struggle with the nuances of social interaction. Standard chatbots tend to be overly verbose, robotic, or purely reactive. The core challenge lies in the requirement for long-horizon planning in social dialogue; an effective companion must steer conversations toward goals like trust-building over multiple turns. Performing Reinforcement Learning (RL) directly on token-level generation is computationally expensive and suffers from "sparse rewards," making it difficult for models to identify which specific words contribute to long-term conversational success.

Methodology

The SAGE framework shifts the focus from token-level generation to high-level "macro-actions" through a multi-stage pipeline.

State-Action Chain (SAC)

Inspired by Chain-of-Thought (CoT) prompting, SAGE augments training data by requiring the model to generate specific "Thinking" tokens before its actual response. These include State Assessment Tokens (e.g., {u_motivation: surprise, u_emotion: excited}) and Dialog Action Tokens (e.g., {a_motivation: curiosity, a_emotion: playful}).

Future-Aware Annotation and Pipeline

A critical innovation in SAGE is Future-Aware Annotation. Rather than labeling turns in isolation, a teacher model (Mixtral 8x7B) analyzes the entire completed conversation to determine states and actions in hindsight, providing a cleaner training signal. The self-improvement pipeline consists of:

  1. SAGE1: Initial fine-tuning on augmented data.
  2. Iterative Tree Search: The model engages in self-play against a user simulator, with an external judge selecting the best trajectories for further tuning (SAGE2, SAGE3).
  3. DPO: Direct Preference Optimization aligns the model with human-like social preferences.
The SAGE data augmentation pipeline, transforming raw dialog into state-action chains.

Key Results

The researchers evaluated SAGE against the original Mixtral 8x7B Instruct model using "LLM-as-a-judge" frameworks (GPT-3.5 and Mixtral).

Improved Social Intelligence and Brevity

SAGE-DPO demonstrated nearly double the win rate against the base Mixtral model in social dialogue. A striking finding was the improvement in conciseness: while the base Mixtral model averaged 86.2 tokens per response, SAGE-DPO averaged only 21.8 tokens. This brevity, combined with a more playful and empathetic tone, makes the AI feel significantly more human.

Comparison of Mixtral 8x7B and SAGE responses, highlighting improvements in tone, brevity, and engagement.

The "Colloquial Tax"

While social performance improved, the model experienced a slight decline in technical benchmarks. For instance, there was a 12.3% drop on GSM8k. This suggests that as models become more colloquial and conversational, they may lose some of the exact-match precision required for formal logic and mathematics.

Discussion and Limitations

Despite its success, SAGE faces certain challenges. The model occasionally suffers from the "Incarnation" Problem, where it forgets it is an AI and attempts to engage in physical human activities (like suggesting a karaoke outing). Additionally, the structured nature of state/action tokens can sometimes lead to formulaic or repetitive responses. Future work will explore hybrid approaches to maintain high-level reasoning capabilities while preserving the improved social intelligence provided by the SAGE framework.

Summary updated

Abstract

Recent advances in large language models have demonstrated impressive capabilities in task-oriented applications, yet building emotionally intelligent chatbots that can engage in natural, strategic conversations remains a challenge. We present a novel approach called SAGE that uses latent variables to control long-horizon behavior in dialogue generation. At the core of our method is the State-Action Chain (SAC), which augments standard language model fine-tuning by introducing latent variables that encapsulate emotional states and conversational strategies between dialogue turns. During inference, these variables are generated before each response, enabling coarse-grained control over dialogue progression while maintaining natural interaction patterns. We also introduce a self-improvement pipeline that leverages dialogue tree search, LLM-based reward modeling, and targeted fine-tuning to optimize conversational trajectories. Our experimental results show that models trained with this approach demonstrate improved performance in emotional intelligence metrics while maintaining strong capabilities on LLM benchmarks. The discrete nature of our latent variables facilitates search-based strategies and provides a foundation for future applications of reinforcement learning to dialogue systems, where learning can occur at the state level rather than the token level. https://github.com/apple/ml-sage-dialog-gen
BibTeX
@misc{zhang2025sage,
  author = {Yizhe Zhang and Navdeep Jaitly},
  title = {{SAGE: Steering and Refining Dialog Generation with State-Action Augmentation}},
  year = {2025},
  eprint = {2503.03040},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.03040},
}
Figure