The Multi-Turn Evaluation Gap in Language Models

84 concepts95 papers3 proposals

Most contemporary research in Large Language Models (LLMs) operates under a "single-turn" paradigm: a model receives a prompt, generates a response, and the interaction ends. While this simplifies benchmarking, it creates a significant disconnect between how models are evaluated and how they are actually used. Across domains ranging from psychological counseling to database querying, researchers are finding that performance on isolated tasks does not necessarily translate to success in the fluid, iterative nature of real-world dialogue.

The Superficiality of Single-Turn Tuning

Instruction Tuning (IT) is often credited with transforming base models into capable assistants. However, Ghosh et al. argue that this process may be more of a "superficial makeover" than a fundamental increase in knowledge. Because IT datasets primarily consist of isolated, single-turn interactions, models often learn to copy patterns rather than engage in deep reasoning. This limitation is particularly evident when models face complex, multi-step requests. Hu et al. observe that standard models frequently ignore steps or fail to complete full sequences when prompts contain multiple distinct instructions. To address this, the authors propose Sequential Instruction Tuning (SIT), which explicitly trains models on "task chains" to bridge the gap between isolated commands and interrelated multi-step reasoning.

Hidden Side Effects and Steerability

Evaluating a model's "steerability"—its ability to follow specific user goals—in a single turn can be misleading. Chang et al. find that current LLMs often exhibit "side effects," where changing one aspect of a text unintentionally alters others. Because these evaluations are typically restricted to single-turn rewriting tasks, they fail to capture how these inconsistencies might compound over an extended conversation. Similarly, Dietz et al. note that while internal auditing personas can detect hidden misbehavior in single-turn interactions, it remains an open question whether these diagnostic signals persist or degrade as the conversational context grows more complex.

The Failure of Static Benchmarks

In specialized fields like psychological counseling or medical advice, the "single-turn" limitation is a matter of safety and efficacy. Traditional benchmarks often rely on static, written exam formats or single-turn Q&A. Wang et al. argue that this fails to account for the development of a "therapeutic alliance," which requires managing evolving dialogue over time. Their framework, CARE-Bench, uses multi-turn client simulations to evaluate models on empathy and clinical skills that static tests cannot measure. Zhang et al. echo this, noting that single-turn datasets like PsyCoPref cannot capture the essential dynamics of professional counseling.

This "evaluation gap" extends to multimodal and technical domains as well:

  • Vision-Language Models: Sinha et al. point out that VQA benchmarks rely on static pairs, leaving a model's ability to handle follow-up queries unmeasured.
  • Database Interfaces: Colas et al. highlight that while data generation for NL-to-QL systems is increasingly multi-turn, the models themselves are still primarily evaluated on single-turn accuracy.
  • Reinforcement Learning: Wu et al. note that theoretical analysis of RLHF is often restricted to single-turn contextual bandits, leaving a gap in our understanding of sequential tasks like long-form conversations.

Moving Toward Narrative and Interaction

The transition from single-turn to multi-turn systems requires a shift in how we think about model outputs. Instead of providing isolated facts, systems may need to act as "storytellers." González-García et al. demonstrate that by identifying semantic relations like causality or contrast, speech assistants can provide contextually rich follow-up information rather than just isolated answers.

However, moving to multi-turn interaction introduces new complexities. Woo et al. note that in full-duplex interactions, reasoning and verbalization must occur concurrently with multiple user inputs, a feat current single-turn frameworks cannot achieve. As Lowe et al. suggest in the context of emergent communication, failing to account for long-term planning and multi-turn exchanges prevents agents from developing the sophisticated communication protocols necessary for human-like interaction.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_single_turn_interaction_limitation_a15e247c,
  title = {single-turn interaction limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-turn-interaction-limitation/rdr_a15e247ca23640efad9c99a0544f7fda},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_single_turn_interaction_limitation_a15e247c_v1,
  title = {single-turn interaction limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-turn-interaction-limitation/rdr_a15e247ca23640efad9c99a0544f7fda/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake