Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in solving isolated coding puzzles on standardized benchmarks, their performance in real-world, iterative software development remains under-explored. Real-world coding is rarely a single-turn event; it involves "tortuous paths" of debugging, refactoring, and evolving requirements. This study addresses the gap in understanding the interaction dynamics of human-LLM collaboration by analyzing over 66,000 real-world coding conversations. We investigate the structural patterns of these dialogues, the models' ability to follow instructions over multiple turns, and the resulting user satisfaction.
Methodology
The research utilizes a multi-stage empirical pipeline designed to extract and analyze high-quality coding data from massive interaction logs.
The pipeline consists of three primary steps:
- Preliminary Data Filtering: Extracting raw code-related data from the LMSYS-Chat-1M and WildChat datasets using rule-based matching.
- Data Disentanglement: Employing an LLM-based approach to "untangle" multi-topic logs into cohesive, single-topic coding tasks, followed by manual inspection to ensure data integrity.
- Empirical Analysis: Modeling dialogues as topological graphs to identify interaction patterns and evaluating them for instruction-following and user satisfaction.
Interaction Pattern Mapping
A core contribution of this work is the classification of coding dialogues into three distinct topological structures: Linear, Star, and Tree patterns.
- Linear (P1): A sequential progression where each turn depends on the previous one.
- Star (P2): A central theme with multiple independent branches of inquiry.
- Tree (P3): A hierarchical expansion where sub-branches emerge from previous turns, often representing complex debugging or design tasks.
Key Insights and Results
Our analysis reveals several critical findings regarding the nature of human-LLM collaboration:
The "4-Turn" Tipping Point and the "Tree of Doom"
Conversations typically remain linear and successful for the first few turns. However, 4 turns marks a critical transition point. Beyond this threshold, interactions frequently shift toward complex "tree" structures. While necessary for complex design, these structures are high-risk: LLMs exhibit a staggering 94.28% non-compliance rate in tree-structured dialogues, struggling to maintain context across deep branches.
Instruction Following and Forgetting
The overall conversation-level accuracy was found to be only 24.07%, indicating that approximately 76% of multi-turn coding chats contain at least one instruction-following failure. Furthermore, LLMs tend to "forget" previous instructions quickly, with a mean step length of roughly 2.65 turns. 59% of these forgetting instances occur within just two steps of the original instruction.
Task-Specific Performance
Performance varies significantly by task type. Database knowledge queries are relatively successful (38.46% failure rate), whereas code generation and refactoring are significantly more challenging, with failure rates of 87.5% and 86.96%, respectively.
Discussion and Future Work
The study highlights a significant decay in user satisfaction as conversations lengthen, particularly during implementation and debugging phases where "generate-revise" loops dominate. While current models like GPT-3.5 and GPT-4 struggle with complex multi-turn structures, these findings provide a roadmap for future development.
We suggest the development of adaptive dialogue systems capable of recognizing high-risk "tree" patterns. Such systems could proactively intervene by summarizing context or re-aligning with the user's original goals to mitigate the high rates of non-compliance and instruction forgetting observed in this study.
Summary updated