GoChat: Goal-oriented Chatbots with Hierarchical Reinforcement Learning

Jianfeng Liu, Feiyang Pan, Ling Luo

Abstract

A chatbot that converses like a human should be goal-oriented (i.e., be purposeful in conversation), which is beyond language generation. However, existing goal-oriented dialogue systems often heavily rely on cumbersome hand-crafted rules or costly labelled datasets, which limits the applicability. In this paper, we propose Goal-oriented Chatbots (GoChat), a framework for end-to-end training the chatbot to maximize the long-term return from offline multi-turn dialogue datasets. Our framework utilizes hierarchical reinforcement learning (HRL), where the high-level policy determines some sub-goals to guide the conversation towards the final goal, and the low-level policy fulfills the sub-goals by generating the corresponding utterance for response. In our experiments conducted on a real-world dialogue dataset for anti-fraud in financial, our approach outperforms previous methods on both the quality of response generation as well as the success rate of accomplishing the goal.

Introduction

Dialogue systems generally fall into two categories: open-domain chatbots, which excel at fluent "chit-chat" but lack direction, and task-oriented systems, which are effective at specific tasks (like booking flights) but often rely on rigid, manual rules. A significant challenge remains in creating chatbots capable of handling multi-turn, strategic conversations where the ultimate goal is only reached at the end of a long interaction. Standard Reinforcement Learning (RL) struggles in these scenarios due to "sparse rewards"—the feedback on success or failure only arrives after many turns, making it difficult to attribute success to specific actions.

Methodology: Hierarchical Reinforcement Learning

GoChat addresses the sparse reward problem using a Hierarchical Reinforcement Learning (HRL) architecture. This framework decomposes the long-term objective into manageable, learnable sub-goals through two primary components:

  1. The Manager (High-level Policy): This component utilizes a Hierarchical Attention Network (HAN) to process the conversation history. Its role is to select a "sub-goal" (a latent category) for each turn, aiming to maximize the long-term reward, such as successfully identifying a scammer.
  2. The Worker (Low-level Policy): Based on the Variational Hierarchical Recurrent Encoder Decoder (VHRED) architecture, the Worker takes the Manager's sub-goal and the current state to generate a specific natural language response.
The GoChat architecture featuring the Manager-Worker hierarchy within an RL environment.

The system employs hybrid rewards: the Manager receives a reward based on the final dialogue outcome, while the Worker receives an internal reward for generating responses that align with the Manager's sub-goals. Sub-goals are discovered in an unsupervised manner using Latent Dirichlet Allocation (LDA) on offline data, identifying 14 distinct dialogue topics.

Experimental Setup

The researchers evaluated GoChat on a real-world anti-fraud dialogue dataset from a large FinTech platform. This dataset contains over 4,000 historical dialogues between security officers and suspected scammers, with an average of 20 turns per dialogue. The only labels provided are binary indicators of whether the final goal (e.g., obtaining evidence) was achieved. To facilitate RL training, the system was first "warm-started" with supervised learning and then fine-tuned using the Advantage Actor-Critic (A2C) algorithm within a simulated environment where a "user model" mimics human responses.

Results and Discussion

GoChat was compared against several state-of-the-art models, including HRED, VHRED, and ReCoSa. As shown in the performance metrics, GoChat achieved superior results across both automated and human evaluations.

Comparison of GoChat against baseline models on language quality and human preference.

Language Quality and Diversity

GoChat achieved the highest BLEU score (9.70) and significantly better diversity scores (Distinct-1: .061, Distinct-2: 0.479), indicating that it generates more varied and human-like text than its predecessors.

Strategic Proactivity

Qualitative analysis reveals that GoChat is more proactive than standard models. Instead of merely reacting to user input, it learns to ask strategic questions to steer the conversation toward its objective. By rewarding the Worker for following sub-goals, the model effectively overcomes the credit assignment problem inherent in long-form dialogue.

Conclusion

GoChat demonstrates that hierarchical decomposition is a powerful tool for bridging the gap between fluent generation and strategic goal-seeking. While the model's performance is currently tied to the quality of the user-model simulator, it represents a significant step toward autonomous, goal-driven conversational agents that can operate without extensive manual rule-engineering. Future work will explore more sophisticated sub-goal discovery methods and the generalizability of the framework to other complex negotiation domains.

Summary updated

Abstract

A chatbot that converses like a human should be goal-oriented (i.e., be purposeful in conversation), which is beyond language generation. However, existing goal-oriented dialogue systems often heavily rely on cumbersome hand-crafted rules or costly labelled datasets, which limits the applicability. In this paper, we propose Goal-oriented Chatbots (GoChat), a framework for end-to-end training the chatbot to maximize the long-term return from offline multi-turn dialogue datasets. Our framework utilizes hierarchical reinforcement learning (HRL), where the high-level policy determines some sub-goals to guide the conversation towards the final goal, and the low-level policy fulfills the sub-goals by generating the corresponding utterance for response. In our experiments conducted on a real-world dialogue dataset for anti-fraud in financial, our approach outperforms previous methods on both the quality of response generation as well as the success rate of accomplishing the goal.
BibTeX
@inproceedings{liu2020gochat,
  author = {Jianfeng Liu and Feiyang Pan and Ling Luo},
  title = {{GoChat: Goal-oriented Chatbots with Hierarchical Reinforcement Learning}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  url = {https://openreview.net/forum?id=qCj8l0xbDH},
}
Figure