Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild

Binquan Zhang, Li Zhang, Haoyuan Zhang, Fang Liu, Song Wang, Bo Shen, An Fu, Lin ShiarXiv 2025 · DOI · arXiv

Introduction

While Large Language Models (LLMs) have demonstrated remarkable proficiency in solving isolated coding puzzles on standardized benchmarks, their performance in real-world, iterative software development remains under-explored. Real-world coding is rarely a single-turn event; it involves "tortuous paths" of debugging, refactoring, and evolving requirements. This study addresses the gap in understanding the interaction dynamics of human-LLM collaboration by analyzing over 66,000 real-world coding conversations. We investigate the structural patterns of these dialogues, the models' ability to follow instructions over multiple turns, and the resulting user satisfaction.

Methodology

The research utilizes a multi-stage empirical pipeline designed to extract and analyze high-quality coding data from massive interaction logs.

Overview of the multi-stage empirical pipeline used to filter, disentangle, and analyze coding-related dialogues.

The pipeline consists of three primary steps:

  1. Preliminary Data Filtering: Extracting raw code-related data from the LMSYS-Chat-1M and WildChat datasets using rule-based matching.
  2. Data Disentanglement: Employing an LLM-based approach to "untangle" multi-topic logs into cohesive, single-topic coding tasks, followed by manual inspection to ensure data integrity.
  3. Empirical Analysis: Modeling dialogues as topological graphs to identify interaction patterns and evaluating them for instruction-following and user satisfaction.

Interaction Pattern Mapping

A core contribution of this work is the classification of coding dialogues into three distinct topological structures: Linear, Star, and Tree patterns.

Topological representations of interaction patterns: Linear (P1), Star (P2), and Tree (P3).
  • Linear (P1): A sequential progression where each turn depends on the previous one.
  • Star (P2): A central theme with multiple independent branches of inquiry.
  • Tree (P3): A hierarchical expansion where sub-branches emerge from previous turns, often representing complex debugging or design tasks.

Key Insights and Results

Our analysis reveals several critical findings regarding the nature of human-LLM collaboration:

The "4-Turn" Tipping Point and the "Tree of Doom"

Conversations typically remain linear and successful for the first few turns. However, 4 turns marks a critical transition point. Beyond this threshold, interactions frequently shift toward complex "tree" structures. While necessary for complex design, these structures are high-risk: LLMs exhibit a staggering 94.28% non-compliance rate in tree-structured dialogues, struggling to maintain context across deep branches.

Instruction Following and Forgetting

The overall conversation-level accuracy was found to be only 24.07%, indicating that approximately 76% of multi-turn coding chats contain at least one instruction-following failure. Furthermore, LLMs tend to "forget" previous instructions quickly, with a mean step length of roughly 2.65 turns. 59% of these forgetting instances occur within just two steps of the original instruction.

Task-Specific Performance

Performance varies significantly by task type. Database knowledge queries are relatively successful (38.46% failure rate), whereas code generation and refactoring are significantly more challenging, with failure rates of 87.5% and 86.96%, respectively.

Discussion and Future Work

The study highlights a significant decay in user satisfaction as conversations lengthen, particularly during implementation and debugging phases where "generate-revise" loops dominate. While current models like GPT-3.5 and GPT-4 struggle with complex multi-turn structures, these findings provide a roadmap for future development.

We suggest the development of adaptive dialogue systems capable of recognizing high-risk "tree" patterns. Such systems could proactively intervene by summarizing context or re-aligning with the user's original goals to mitigate the high rates of non-compliance and instruction forgetting observed in this study.

Summary updated

BibTeX
@misc{zhang2025decoding,
  author = {Binquan Zhang and Li Zhang and Haoyuan Zhang and Fang Liu and Song Wang and Bo Shen and An Fu and Lin Shi},
  title = {{Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild}},
  year = {2025},
  eprint = {2512.10493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.10493},
}
Figure