Phrase-Level Action Reinforcement Learning for Neural Dialog Response Generation

Takato Yamazaki, Akiko AizawaACL 2021

TL;DR

This paper introduces PHRASERL, a novel approach to neural dialog response generation that uses phrases as discrete actions for reinforcement learning. By breaking down responses into manageable phrase-level units, PHRASERL achieves more flexible and human-like dialog generation, outperforming previous state-of-the-art methods on the MultiWOZ dataset.

What Problem Does This Paper Address?

Generating natural and effective responses in task-oriented dialog systems is a complex challenge. Traditional methods often rely on defining a rigid action space for reinforcement learning (RL), which can lead to responses that are either grammatically awkward or lack task-specific information. Previous attempts to address this, like using discrete latent variables for entire dialog acts, have struggled to balance flexibility and control. The core problem is finding an action space that is expressive enough to capture the nuances of human language and task completion, yet structured enough for efficient RL optimization.

Methods

The paper proposes Phrase-Level Action Reinforcement Learning (PHRASERL). The core of PHRASERL is a neural Hidden Semi-Markov Model (HSMM) decoder. This HSMM decoder is trained to generate "typed phrases" from hidden states. These typed phrases then form the discrete action space for the RL policy.

Here's a breakdown of the key methodological components:

  • HSMM Decoder: This model treats sentences as sequences of phrases, where each hidden state in the HSMM corresponds to a phrase. It learns to generate these phrases based on a source input (contextual information like dialog history and belief state).
  • Phrase as Action: Instead of treating individual words or entire dialog acts as actions, PHRASERL uses the phrases generated by the HSMM decoder as the discrete actions. This allows the RL agent to learn to select and reorder these phrases to construct a complete response.
  • Supervised Pre-training: The HSMM decoder is first trained using supervised learning to generate useful phrases.
  • Reinforcement Learning Fine-tuning: After pre-training, the model is fine-tuned using RL. The RL policy learns to select sequences of hidden states (phrases) to maximize task success and response quality.
  • Discrete Embeddings for Context: To mitigate the "interdependence problem" (where a powerful encoder might make the decoder overly reliant on context and ignore hidden states), the authors use discrete embeddings for contextual information, achieved through straight-through Gumbel-softmax.

Datasets

The experiments were conducted on the MultiWOZ dataset. This is a large-scale, multi-domain wizard-of-oz dataset for task-oriented dialogue modeling, covering seven domains like booking restaurants and hotels. The authors used the Dialogue-Context-to-Text Generation task from the original paper.

Key Insights and Ideas

  • Phrases as a Sweet Spot for Action Spaces: The paper demonstrates that using phrases as discrete actions for RL offers a better balance between expressiveness and tractability compared to word-level actions or global dialog act vectors.
  • Disentangled Phrase Generation: The HSMM decoder learns to generate "typed phrases," meaning hidden states tend to represent specific types of phrases (e.g., "verb phrases for the end of questions," "features of a facility"). This disentanglement is crucial for flexible response construction.
  • Discrete Embeddings Improve RL: Using discrete embeddings for contextual information significantly improved the performance of the RL agent, suggesting that reducing the expressiveness of the context encoder helps the RL policy learn more effectively.
  • Flexibility in Response Generation: The phrase-level action space allows the model to construct diverse and contextually appropriate responses by reordering learned phrases, leading to better informativeness and grammatical fluency.

Key Results

  • Competitive Performance: PHRASERL (specifically the DISC-RL variant) achieved competitive results with state-of-the-art models on the MultiWOZ dataset, outperforming previous RL-based methods like LaRL in BLEU score.
  • Improved Inform Rate and Success Rate: The model showed significant improvements in Inform Rate and Success Rate, indicating its ability to convey necessary information and complete tasks effectively.
  • Discrete Embeddings Outperform Continuous: The discrete embedding version of PHRASERL consistently outperformed its continuous embedding counterpart across all metrics and showed a sharper reward increase during RL training.
  • Grammaticality and Informativeness: Qualitative analysis showed that PHRASERL can generate grammatical and informative responses, often including details like hotel names and phone numbers, which are rewarded during RL training.

Limitations and Future Work

  • Grammar Drawback: While PHRASERL achieves good results, the paper notes that using fixed phrases and arranging them can still have a drawback in generating perfectly grammatical responses due to a lack of word-level flexibility.
  • Hierarchical RL: The authors suggest a future direction of applying hierarchical reinforcement learning, where a high-level policy determines the dialog act of the entire response, and a low-level policy designs the syntax of the response.
  • External Rewards: While PHRASERL shows some resistance to degeneration without external modules, further improvements could be expected by incorporating external rewards for grammaticality.

Summary updated

BibTeX
@inproceedings{yamazaki2021phrase,
  author = {Takato Yamazaki and Akiko Aizawa},
  title = {{Phrase-Level Action Reinforcement Learning for Neural Dialog Response Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure