DyBBT: Dynamic Balance via Bandit inspired Targeting for Dialog Policy with Cognitive Dual-Systems

Shuyu Zhang, Yifan Wei, Jialuo Yuan, Xinru Wang, Yanmin Zhu, Bin Li

TL;DR

DyBBT (Dynamic Balance via Bandit-inspired Targeting) is a novel framework for task-oriented dialog that mimics human cognition by switching between a fast, intuitive "System 1" and a slow, deliberative "System 2." By navigating a structured 3D "cognitive state space" using bandit-inspired logic, the system achieves state-of-the-art performance and exploration efficiency across complex multi-domain benchmarks.

The Exploration-Exploitation Dilemma in Dialog

Task-oriented dialog systems (TODS)—such as those for booking services—face a fundamental "exploration-exploitation" dilemma. The system must decide whether to stick to known successful strategies (exploitation) or try new approaches to improve user assistance (exploration).

Traditional Deep Reinforcement Learning (DRL) methods often rely on static exploration heuristics, like $\epsilon$-greedy, which fail to adapt to the dialog context. For instance, a system should explore more when user intent is ambiguous early in a conversation but focus on precision when finalizing critical booking details. While Large Language Model (LLM) policies can reason deeply, they are often too computationally expensive for every turn. DyBBT addresses this by creating a policy that dynamically perceives "cognitive affordances"—the specific needs for exploration or deliberation—to balance efficiency and robustness.

Comparison of exploration strategies: Traditional DRL (static), EIERL (population-based), and DyBBT (cognitive dual-system).

Methodology: The Cognitive Dual-System

The core of DyBBT is a Cognitive Dual-System architecture managed by a bandit-inspired meta-controller.

1. Cognitive State Space ($C$)

Rather than processing high-dimensional belief states directly, DyBBT compresses the dialog context into three interpretable dimensions:

  • Dialog Progress ($d_t$): Proximity to dialog completion.
  • User Uncertainty ($u_t$): The entropy or ambiguity in user goals.
  • Slot Dependency ($\rho_t$): The degree to which the next action depends on previously acquired information.

2. Dual Systems

  • System 1 (Fast): A lightweight LLM fine-tuned (via SFT and PPO) for routine, low-latency responses.
  • System 2 (Slow): The same backbone coupled with multi-path reasoning to handle novel or complex states.
The DyBBT Cognitive Dual-System architecture, balancing fast intuitive inference (System 1) with slow deliberative reasoning (System 2) based on cognitive state.

3. Meta-Controller and Knowledge Distillation

A meta-controller, formulated as a Contextual Multi-Armed Bandit (CMAB), decides when to trigger System 2. It invokes the deliberative system if the current cognitive state has low visitation (Upper Confidence Bound logic) or if System 1 reports low confidence. High-quality decisions from System 2 are distilled back into System 1, allowing the "intuitive" system to improve over time and reducing future reliance on expensive deliberation.

Evaluation and Key Results

The authors evaluated DyBBT on the Microsoft Dialog Challenge (Movie, Restaurant, Taxi) and the large-scale MultiWOZ 2.1 benchmark.

  • State-of-the-Art Performance: DyBBT achieved the highest success rates across all single-domain tasks. On MultiWOZ, DyBBT-8B reached an 84.1% success rate, significantly outperforming the previous SOTA, EIERL (18.5% in multi-domain settings).
  • Efficiency Gains: DyBBT-8B/GPT-4.0 matched pure GPT-4.0 performance with a 2.3x speedup in inference time and lower costs, as System 2 was only invoked ~14% of the time.
  • Human Alignment: DyBBT’s decisions to "stop and think" aligned with human expert judgment 88.7% of the time.

Insights and Limitations

DyBBT bridges theory and practice by mapping "affordances" from ecological psychology to quantifiable dialog features. This allows bandit theory to be applied to a tractable 3D space, achieving sublinear regret and efficient learning.

However, the framework currently relies on hand-crafted features for the cognitive state space. Future work could involve learning these representations end-to-end. Additionally, the system's performance is sensitive to the quantization of the cognitive space and the quality of System 2, as errors in deliberation can be distilled into the intuitive policy.

Summary updated

BibTeX
@inproceedings{zhang2026dybbt,
  author = {Shuyu Zhang and Yifan Wei and Jialuo Yuan and Xinru Wang and Yanmin Zhu and Bin Li},
  title = {{DyBBT: Dynamic Balance via Bandit inspired Targeting for Dialog Policy with Cognitive Dual-Systems}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2180},
}
Figure