GPT-Critic: Offline Reinforcement Learning for End-to-End Task-Oriented Dialogue Systems

Youngsoo Jang, Jongmin Lee, Kee-Eung KimICLR 2022 · OpenReview

TL;DR

GPT-Critic is an offline reinforcement learning (RL) framework that allows task-oriented dialogue systems to improve their goal-reaching strategies without losing their ability to speak natural human language. By using a "critic" to select the best self-generated responses and then fine-tuning the model on this improved data, it achieves state-of-the-art performance on major benchmarks like MultiWOZ 2.0.

What Problem Does This Paper Address?

Building a task-oriented dialogue system (like a virtual travel agent) requires balancing two difficult goals: task success (booking the right hotel) and naturalness (sounding like a human).

Most current systems use Imitation Learning, where they simply copy human-to-human dialogue datasets. However, these datasets are "noisy"—humans often make mistakes, fail to solve the task, or provide suboptimal paths. While we could use Reinforcement Learning (RL) to optimize for success, standard RL in the world of natural language is notoriously unstable. Because the "action space" (every possible sentence) is nearly infinite, RL agents often learn "shortcuts"—generating repetitive or nonsensical gibberish that technically maximizes a reward but is useless to a human user. This is known as language divergence.

The authors tackle the challenge of Offline RL: how can we take a static, imperfect dataset and train an agent to be better than the humans in that data, all while ensuring it remains coherent and fluent?

Methods

GPT-Critic is built on top of GPT-2 and follows an iterative "Actor-Critic" style approach, but with a clever twist called Dataset Revision.

  1. The Critic: The authors train a GPT-2-based "Critic" to estimate the value ($Q$-value) of a specific response given the dialogue history. It learns to predict which responses lead to a successful task completion using temporal difference learning on the static dataset.
  2. Critic-Guided Self-Generation: Instead of updating the policy via risky gradients, the system uses the current policy to generate $N$ candidate responses for every turn in the training data. The Critic then evaluates these candidates and picks the one with the highest predicted value.
  3. Dataset Revision: The original (potentially flawed) human responses in the dataset are replaced with these "Critic-approved" self-generated responses. This creates a new, "revised" version of the dataset that is strategically superior but still linguistically grounded.
Overview of the GPT-Critic framework
  1. Behavior Cloning (BC): The policy is then fine-tuned on this revised dataset using standard supervised learning. Because the model is imitating sentences it generated itself (which were originally seeded by a pre-trained language model), it stays within the bounds of natural human language.

This process can be repeated iteratively, and the authors provide a mathematical proof (the Policy Improvement Theorem) showing that this method consistently leads to a better policy.

Datasets

The researchers evaluated GPT-Critic on two primary benchmarks:

  • MultiWOZ 2.0: The most widely used multi-domain task-oriented dialogue dataset, containing over 8,000 dialogues across domains like hotels, taxis, and restaurants.
  • ConvLab: An open-source framework used for simulator-based evaluation, allowing the agent to interact with a rule-based user simulator to test its robustness in dynamic scenarios.

Key Insights and Ideas

  • Revision over Filtering: Traditional "Weighted Behavior Cloning" simply throws away unsuccessful dialogues. GPT-Critic is more efficient; it keeps the useful context of a "failed" dialogue but revises the agent's responses to turn that failure into a success.
  • Stability via Imitation: By turning an RL problem back into a supervised learning (BC) problem on a revised dataset, the authors bypass the instability of policy gradients. This acts as a "naturalness" constraint.
  • The Power of Candidates: Even with a small number of samples ($N=5$), the model can find significantly better strategic paths than those present in the original human data.
  • Avoiding the "OOD" Trap: Offline RL usually struggles with "Out-of-Distribution" (OOD) actions. GPT-Critic mitigates this by only evaluating actions that the current policy is actually capable of generating.

Key Results

  • New State-of-the-Art: On MultiWOZ 2.0, GPT-Critic achieved a Combined Score of 101.13, significantly outperforming the previous leader, UBAR (98.56).
Performance comparison on the MultiWOZ 2.0 benchmark
  • Higher Success Rates: In simulator-based tests (ConvLab), GPT-Critic reached a 77.70% success rate, compared to 74.30% for UBAR and 72.60% for other offline RL methods like CRR.
  • Human Preference: In Amazon Mechanical Turk evaluations, humans rated GPT-Critic significantly higher for Appropriateness than all baselines, while its Fluency remained indistinguishable from models trained purely on human data.
  • Qualitative Superiority: While standard RL (like REINFORCE) eventually devolved into generating repetitive nonsense (e.g., "ask ask ask ask..."), GPT-Critic maintained clear, helpful, and diverse responses.

Limitations and Future Work

  • Computational Cost: The "Dataset Revision" step is computationally expensive, as it requires generating multiple candidates for every single turn in the training corpus.
  • Reward Function Reliance: Like all RL, the method depends on a well-defined reward function. In many real-world scenarios, defining "success" is more ambiguous than in the MultiWOZ benchmark.
  • Efficiency vs. Success: The authors noted that GPT-Critic sometimes takes slightly more dialogue turns to complete a task. It prioritizes getting it right over getting it done fast, which might be a trade-off depending on the specific application.
  • Future Direction: The authors suggest exploring how this method scales with even larger language models (like GPT-3 or beyond) and more complex, multi-modal dialogue tasks.

Summary updated

Abstract

Training a task-oriented dialogue agent can be naturally formulated as offline reinforcement learning (RL) problem, where the agent aims to learn a conversational strategy to achieve user goals, only from a dialogue corpus. It is very challenging in terms of RL since the natural language action space is astronomical, while feasible (syntactically and semantically correct) actions are very sparse. Thus, standard RL methods easily fail and generate responses diverging from human language, even when fine-tuning a powerful pre-trained language model. In this paper, we introduce GPT-Critic, an offline RL method for task-oriented dialogue. GPT-Critic is built upon GPT-2, fine-tuning the language model through behavior cloning of the critic-guided self-generated sentences. GPT-Critic is essentially free from the issue of diverging from human language since it learns from the sentences sampled from the pre-trained language model. In the experiments, we demonstrate that our algorithm outperforms the state-of-the-art in the task-oriented dialogue benchmarks including MultiWOZ 2.0 and ConvLab.
BibTeX
@inproceedings{jang2022gpt,
  author = {Youngsoo Jang and Jongmin Lee and Kee-Eung Kim},
  title = {{GPT-Critic: Offline Reinforcement Learning for End-to-End Task-Oriented Dialogue Systems}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=qaxhBG1UUaS},
}
Figure