ConsistentChat: Building Skeleton-Guided Consistent Dialogues for Large Language Models from Scratch

Jiawei Chen, Xinyan Guan, Qianhao Yuan, Guozhao Mo, Weixiang Zhou, Yaojie Lu, Hongyu Lin, Ben He, Le Sun, Xianpei HanarXiv 2025 · DOI · arXiv

Abstract

While Large Language Models (LLMs) excel at single-turn instruction following, they frequently suffer from "context drift" during extended multi-turn conversations, leading to a loss of semantic coherence and failure to adhere to initial constraints. This paper introduces a novel "Skeleton-Guided" framework designed to synthesize high-quality multi-turn training data by modeling human conversational intents and global information flow. By generating complete dialogue skeletons and utilizing a single-pass response generation strategy, the resulting ConsistentChat dataset enables models to maintain strict consistency across deep dialogue turns. Our results demonstrate that models fine-tuned on ConsistentChat not only achieve superior multi-turn performance but also exhibit significant improvements in general reasoning and coding capabilities.

Introduction: The Challenge of Multi-turn Consistency

Current LLM training paradigms rely heavily on synthetic instruction datasets like Self-Instruct or Alpaca, which are predominantly composed of single-turn exchanges. When extended to multi-turn scenarios, these models often degrade, forgetting earlier instructions or drifting off-topic. Existing multi-turn datasets frequently exacerbate this by stitching together local turns without a global plan, failing to capture the goal-oriented progression of natural human dialogue.

Preliminary investigations reveal that even state-of-the-art models experience a steady decline in instruction-following ability as conversations deepen, particularly after the sixth turn. To address this, we propose a framework that moves beyond reactive, turn-by-turn generation toward a proactive, globally planned dialogue synthesis.

Methodology: Skeleton-Guided Synthesis

We introduce a Skeleton-Guided Multi-Turn Dialogue Generation framework that operates in two distinct stages to ensure global coherence:

  1. Intent and Flow Modeling: Drawing from human-computer interaction theories, we define nine fundamental conversational intents (e.g., Problem-Solving, Emotional Support, Educational). Each intent is mapped to a specific "information flow," such as transitioning from problem diagnosis to solution optimization.
  2. Skeleton-to-Dialogue Generation: Unlike traditional methods that generate turns sequentially, our framework first prompts an LLM to generate a complete sequence of user queries (the "skeleton") that follows the logical flow of the chosen intent. Subsequently, the model generates all agent responses in a single forward pass using Chain-of-Thought (CoT) reasoning.
Comparison of single-turn and multi-turn instruction synthesis manners, highlighting the ConsistentChat approach.

This single-pass approach effectively bypasses the "causal masking" limitation of LLMs. By providing the model with the entire planned sequence of user queries at once, it can generate responses that are globally consistent and proactively anticipate future turns, rather than merely reacting to the immediate prompt.

The ConsistentChat Dataset

Using this framework, we constructed ConsistentChat, a dataset containing approximately 15,000 multi-turn conversations and over 224,000 utterances. The data spans the nine defined intent categories, providing a diverse and structured training signal for conversational consistency.

Experimental Results

We evaluated models fine-tuned on ConsistentChat against several baselines, including ShareGPT and UltraChat, across benchmarks such as LIGHT (character consistency), TOPDIAL (proactive dialogue), and MT-EVAL (comprehensive multi-turn tasks).

Performance and Consistency

Models trained on ConsistentChat showed a 20–30% improvement in consistency scores on the LIGHT and TOPDIAL benchmarks. Notably, a Qwen-2.5-7B model fine-tuned on our dataset outperformed the much larger Qwen-2.5-14B-Instruct model in multi-turn evaluations, suggesting that data quality and coherence are more critical than parameter count for conversational stability.

Performance scores across dialogue turns, illustrating the degradation in refinement tasks and the impact of switching tasks.

General Reasoning Gains

Perhaps most surprisingly, the structured nature of ConsistentChat translated to broader cognitive improvements. The Qwen-2.5-7B model saw a 30.4% increase in MATH scores and a 33.7% increase in HumanEval (coding) scores compared to its base version. This suggests that learning to maintain logical flow in dialogue reinforces the underlying reasoning capabilities of the model.

Conclusion

The ConsistentChat framework demonstrates that conversational consistency is a product of data structure rather than just scale. By guiding synthesis with human-like intent skeletons, we can produce models that remain coherent throughout deep dialogues. Future work will involve expanding the range of conversational intents and exploring the scalability of this framework to frontier-class models.

Summary updated

Abstract

Current instruction data synthesis methods primarily focus on single-turn instructions and often neglect cross-turn coherence, resulting in context drift and reduced task completion rates in extended conversations. To address this limitation, we propose Skeleton-Guided Multi-Turn Dialogue Generation, a framework that constrains multi-turn instruction synthesis by explicitly modeling human conversational intent. It operates in two stages: (1) Intent Modeling, which captures the global structure of human dialogues by assigning each conversation to one of nine well-defined intent trajectories, ensuring a coherent and goal-oriented information flow; and (2) Skeleton Generation, which constructs a structurally grounded sequence of user queries aligned with the modeled intent, thereby serving as a scaffold that constrains and guides the downstream instruction synthesis process. Based on this process, we construct ConsistentChat, a multi-turn instruction dataset with approximately 15,000 multi-turn conversations and 224,392 utterances. Experiments on the Light, Topdial, and MT-Eval benchmarks show that models fine-tuned on ConsistentChat achieve a 20-30% improvement in chat consistency and up to a 15% increase in task success rate, significantly outperforming models trained on existing single-turn and multi-turn instruction datasets.
BibTeX
@misc{chen2025consistentchat,
  author = {Jiawei Chen and Xinyan Guan and Qianhao Yuan and Guozhao Mo and Weixiang Zhou and Yaojie Lu and Hongyu Lin and Ben He and Le Sun and Xianpei Han},
  title = {{ConsistentChat: Building Skeleton-Guided Consistent Dialogues for Large Language Models from Scratch}},
  year = {2025},
  eprint = {2506.03558},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.03558},
}
Figure