Abstract
SAGE (State Augmented GEneration) is a framework designed to enhance the social intelligence and strategic planning of conversational agents. By introducing a "State-Action Chain" (SAC), the model predicts latent variables—user emotional states and intended bot strategies—prior to response generation. This approach results in dialogue that is more concise, empathetic, and human-like compared to standard instruction-tuned Large Language Models (LLMs).
Introduction
While Large Language Models (LLMs) excel at task-oriented goals like coding or mathematical reasoning, they often struggle with the nuances of social interaction. Standard chatbots tend to be overly verbose, robotic, or purely reactive. The core challenge lies in the requirement for long-horizon planning in social dialogue; an effective companion must steer conversations toward goals like trust-building over multiple turns. Performing Reinforcement Learning (RL) directly on token-level generation is computationally expensive and suffers from "sparse rewards," making it difficult for models to identify which specific words contribute to long-term conversational success.
Methodology
The SAGE framework shifts the focus from token-level generation to high-level "macro-actions" through a multi-stage pipeline.
State-Action Chain (SAC)
Inspired by Chain-of-Thought (CoT) prompting, SAGE augments training data by requiring the model to generate specific "Thinking" tokens before its actual response. These include State Assessment Tokens (e.g., {u_motivation: surprise, u_emotion: excited}) and Dialog Action Tokens (e.g., {a_motivation: curiosity, a_emotion: playful}).
Future-Aware Annotation and Pipeline
A critical innovation in SAGE is Future-Aware Annotation. Rather than labeling turns in isolation, a teacher model (Mixtral 8x7B) analyzes the entire completed conversation to determine states and actions in hindsight, providing a cleaner training signal. The self-improvement pipeline consists of:
- SAGE1: Initial fine-tuning on augmented data.
- Iterative Tree Search: The model engages in self-play against a user simulator, with an external judge selecting the best trajectories for further tuning (SAGE2, SAGE3).
- DPO: Direct Preference Optimization aligns the model with human-like social preferences.
Key Results
The researchers evaluated SAGE against the original Mixtral 8x7B Instruct model using "LLM-as-a-judge" frameworks (GPT-3.5 and Mixtral).
Improved Social Intelligence and Brevity
SAGE-DPO demonstrated nearly double the win rate against the base Mixtral model in social dialogue. A striking finding was the improvement in conciseness: while the base Mixtral model averaged 86.2 tokens per response, SAGE-DPO averaged only 21.8 tokens. This brevity, combined with a more playful and empathetic tone, makes the AI feel significantly more human.
The "Colloquial Tax"
While social performance improved, the model experienced a slight decline in technical benchmarks. For instance, there was a 12.3% drop on GSM8k. This suggests that as models become more colloquial and conversational, they may lose some of the exact-match precision required for formal logic and mathematics.
Discussion and Limitations
Despite its success, SAGE faces certain challenges. The model occasionally suffers from the "Incarnation" Problem, where it forgets it is an AI and attempts to engage in physical human activities (like suggesting a karaoke outing). Additionally, the structured nature of state/action tokens can sometimes lead to formulaic or repetitive responses. Future work will explore hybrid approaches to maintain high-level reasoning capabilities while preserving the improved social intelligence provided by the SAGE framework.
Summary updated