Multi-Action Dialog Policy Learning from Logged User Feedback

Shuo Zhang, Junzhou Zhao, Pinghui Wang, Tianxiang Wang, Zi Liang, Jing Tao, Yi Huang, Junlan Feng

Introduction

Task-oriented dialog systems, designed to assist users with specific goals such as booking flights or finding restaurants, increasingly rely on "multi-action" policies. These policies allow an agent to perform multiple communicative acts—such as confirming a selection while simultaneously requesting missing information—within a single conversational turn. However, training these complex policies effectively remains a significant hurdle. Traditional Supervised Learning (SL) requires exhaustive expert-labeled datasets, while Reinforcement Learning (RL) often suffers from sparse, delayed, and noisy rewards.

This paper introduces BanditMatch, a novel framework designed to bridge this gap by leveraging "logged user feedback." By treating dialog policy learning as a feedback-enhanced semi-supervised learning problem, BanditMatch utilizes naturally occurring turn-level interactions to refine agent behavior without the need for continuous expert intervention.

The Challenge: Partial Feedback in Multi-Action Spaces

In a multi-action setting, the action space is combinatorial, making it difficult for agents to identify the "optimal" response from thousands of possibilities. While logged user feedback (e.g., a user stating "That's not what I asked for") provides a signal, it is inherently partial. Negative feedback indicates a failure but does not explicitly point to the correct alternative. Conversely, positive feedback confirms a specific path but does not explore other potentially more efficient ones.

The BanditMatch Framework

BanditMatch addresses these challenges through a three-pronged architecture that combines semi-supervised learning with bandit algorithms:

1. Feedback-Enhanced Thresholding (FET)

Standard semi-supervised learning models often use static confidence thresholds to generate "pseudo-labels" from unlabeled data. BanditMatch introduces FET, which dynamically adjusts these thresholds for every action class. By analyzing the model’s performance on instances with positive user feedback, FET scales thresholds based on the inherent learning difficulty of specific actions (e.g., distinguishing between a simple "greeting" and a complex "offer" with multiple attributes).

2. Semi-Supervised Learning (SSL) with Consistency Regularization

For high-confidence examples identified by FET, the framework employs a modified FixMatch approach. It utilizes "Mix-up" data augmentation, blending different dialog states to ensure the model's predictions remain robust and consistent across similar conversational contexts. This helps the model generalize from a small set of expert labels to a larger pool of unlabeled interactions.

3. Counterfactual Bandit Learning (CBL)

To handle "unconfident" examples where the model is uncertain, BanditMatch incorporates Counterfactual Risk Minimization (CRM). This allows the agent to learn from its mistakes by estimating the potential rewards of actions it did not take. By partitioning data based on feedback and confidence, the model can effectively explore the action space and avoid repeating documented errors.

To maintain linguistic coherence, a KL Control mechanism is integrated, ensuring the learned policy does not deviate so drastically from the initial logging policy that it produces nonsensical or unrealistic action combinations.

Experimental Evaluation

The researchers evaluated BanditMatch using the MultiWOZ 2.0 dataset, a standard benchmark for multi-domain task-oriented dialog. To simulate real-world constraints, the model was initialized with a "logging policy" trained on only 10% of the available expert-labeled data. The remaining data was treated as unlabeled, with feedback generated via an agenda-based user simulator.

Key Results

  • Superior Success Rates: BanditMatch achieved a success rate of 76.7%, significantly outperforming the "Full Supervised Learning" baseline (67.3%) and specialized bandit methods like BanditNet (74.3%).
  • Improved Conciseness: The model showed a marked improvement in Inform F1 scores (a 9% to 23% increase over baselines). This indicates that the agent learned to provide precise information, reducing redundancy and "wordiness" in its responses.
  • Learning Stability: Unlike standard RL algorithms such as PPO, which can be unstable when rewards are sparse, BanditMatch demonstrated steady, monotonic improvement even with minimal expert data.

Limitations and Future Directions

While BanditMatch represents a significant step forward, the authors note several areas for future research. Currently, the framework assumes user feedback is perfectly accurate, which is rarely the case in human-agent interaction. Future iterations could incorporate "unbiased estimation" techniques to handle noisy or contradictory feedback. Additionally, the authors suggest exploring Federated Bandit Learning to process user feedback locally on devices, thereby enhancing data privacy. Finally, addressing the "cold start" problem—where the initial logging policy is too weak to generate useful feedback—remains a critical challenge for deployment.

Summary updated

BibTeX
@inproceedings{zhang2023multi,
  author = {Shuo Zhang and Junzhou Zhao and Pinghui Wang and Tianxiang Wang and Zi Liang and Jing Tao and Yi Huang and Junlan Feng},
  title = {{Multi-Action Dialog Policy Learning from Logged User Feedback}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26636},
}
Figure