Overcoming Myopic Planning in Multi-Turn Dialogue

20 concepts23 papers2 proposals

Most Large Language Models (LLMs) are trained as "passive responders." Because their primary training objectives—next-token prediction and standard Reinforcement Learning from Human Feedback (RLHF)—focus on maximizing the quality of the immediate next response, they often fail to achieve long-term goals that require strategic coordination over multiple turns. This phenomenon, often described as myopic planning, prevents agents from effectively navigating complex interactions like negotiation, pedagogy, or emotional support.

The Single-Turn Optimization Bias

The core of the problem lies in a granularity mismatch. Standard alignment techniques like Direct Preference Optimization (DPO) typically operate on a single-turn basis. As Shaikh et al. demonstrate, this focus can actually widen the "grounding gap"—the tendency for models to skip essential clarifying questions and acknowledgments in favor of direct, immediate answers. In high-stakes dialogues like therapy or teaching, this lack of grounding prevents the coordination necessary for mutual understanding.

When a model is rewarded only for the current turn, it lacks the incentive to take "bridge actions"—moves that might not offer immediate utility but are essential for reaching a successful outcome later. For instance, in emotional support contexts, Cheng et al. note that systems often select strategies based solely on the current context, ignoring how those choices might influence the user’s emotional state several turns down the line.

Hierarchical Planning and Credit Assignment

To move beyond reactive behavior, researchers have turned to Hierarchical Reinforcement Learning (HRL). The intuition is to decouple high-level strategy from low-level text generation. In the GoChat framework, Liu et al. use a "Manager" to set sub-goals and a "Worker" to generate fluent responses, allowing the agent to proactively steer the interaction toward a long-term objective.

However, multi-turn RL faces a massive credit-assignment challenge: if a conversation fails after ten turns, which specific word or sentence was responsible? Zhou et al. address this with ArCHer, a framework that splits learning into "utterance-level" planning and "token-level" generation. By rewarding high-level strategic choices rather than just individual tokens, they achieve significantly more efficient learning for complex tasks like web navigation or detective games.

Simulating the Future

Another approach to breaking myopia is to give the model "foresight" during the generation process. Rather than just looking at the past context, models can simulate potential future turns to evaluate the long-term impact of their current words.

  • Lookahead Heuristics: Liu et al. introduced ProphetChat, which simulates potential user reactions during inference to generate more engaging and informative responses.
  • Semantic Space Search: To avoid the high latency of simulating full text, Chen et al. proposed SCOPE, which performs Monte Carlo Tree Search (MCTS) within a learned embedding space. This allows the model to plan several steps ahead 70x faster than standard simulation methods.
  • Active Collaboration: Shirley Wu developed CollabLLM, which uses "Multiturn-aware Rewards" derived from simulated future interactions. This trains the model to proactively clarify ambiguous user intents rather than providing a generic, immediate answer.

Aligning for Global Success

Recent work has also reimagined the alignment process itself to account for multi-turn dependencies. Instead of comparing individual responses, Multi-turn Preference Optimization (MTPO), proposed by Shani et al., aligns agents by comparing entire conversational trajectories. This treats dialogue as a strategic game, seeking a Nash equilibrium where the agent's moves are optimized for the final outcome of the interaction.

Similarly, Kong et al. introduced Segment-Level DPO (SDPO), which identifies and optimizes key multi-turn segments rather than isolated turns. By focusing on the flow of the conversation, these methods allow open-source models to develop the "social intelligence" required to influence user beliefs and achieve complex goals, such as persuasion in social deduction games (Ye et al.) or refined product recommendations (Feng et al.).

By shifting the optimization target from the "next turn" to the "final goal," these frameworks enable LLMs to transition from passive text generators to strategic conversational partners.

Key Foundational Work

GoChat: Goal-oriented Chatbots with Hierarchical Reinforcement Learning (2020) — contributed 1 concept elements to this direction.
Alana: Social Dialogue using an Ensemble Model and a Ranker trained on User Feedback (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_single_turn_optimization_bias_32d04ae4,
  title = {single-turn optimization bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-turn-optimization-bias/rdr_32d04ae461bc446daec2ff44af087494},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_single_turn_optimization_bias_32d04ae4_v1,
  title = {single-turn optimization bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-turn-optimization-bias/rdr_32d04ae461bc446daec2ff44af087494/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake