Pretrain-Finetune Based Training of Task-Oriented Dialogue Systems in a Real-World Setting

Manisha Srivastava, Yichao Lu, Riley Peschon, Chenyang LiNAACL 2021 · DOI

TL;DR

This paper tackles the challenge of limited training data in task-oriented dialogue systems by proposing a pretrain-finetune approach. They demonstrate that pretraining on a large, noisy, unannotated dataset significantly boosts performance, and a subsequent finetuning step with a small, high-quality dataset further improves results, achieving impressive real-world performance in customer service.

What Problem Does This Paper Address?

Building effective task-oriented dialogue systems, which are designed to help users accomplish specific goals (like booking a flight or canceling an order), is heavily reliant on large amounts of high-quality, annotated training data. However, creating such datasets is expensive and time-consuming. Existing synthetic datasets often fail to capture the nuances of real-world conversations. This scarcity of data makes it difficult to train accurate models from scratch, leading to systems that struggle with the complexities and variations of human interaction.

Methods

The core of this paper's contribution lies in its pretrain-finetune methodology.

  1. Pretraining: They leverage a large, unannotated dataset of historical customer service chats. The model is pretrained using a next sentence prediction task. This involves training the model to predict whether a given agent response is appropriate for a given dialogue context. To create training examples, true agent responses form positive pairs, while randomly selected responses form negative pairs. This phase aims to imbue the model with a general understanding of conversational flow and appropriateness.
  2. Finetuning: After pretraining, the model is finetuned on a smaller, high-quality, annotated dataset. This dataset is collected under controlled conditions with specialist agents to ensure accuracy and completeness.
  3. Regularization: To prevent the model from "forgetting" the knowledge gained during pretraining (a common issue in finetuning), they employ a data-mix regularization technique. This involves interleaving batches from the pretraining dataset with batches from the finetuning dataset during training.
  4. Loss Function: They explore two loss functions:
  5. Binary Cross-Entropy (BCE): A standard classification loss.
  6. Ranking Loss: A novel loss function designed to better capture the relative scores of candidate responses, which is crucial for retrieval-based systems where the top-ranked response is chosen. This loss directly minimizes the negative log probability of the true response given the context, considering the scores of other potential responses.

Datasets

  • Pretraining Dataset: A large dataset (hundreds of thousands of chats) of historical customer service chat transcripts for a delivery-related issue. This data is described as "noisy" and "unannotated" in terms of specific annotations but contains dialogue turns.
  • Finetuning Dataset: A smaller dataset (a few thousand chats) for the same delivery-related issue, collected by specialist agents with accurate and complete annotations (customer profile, order details). This dataset also includes a template pool of 85 responses.
  • Evaluation Datasets: Two test sets are used:
  • Pretrain Test Dataset: Historical chat transcripts from a period not overlapping with the pretraining training data.
  • Finetune Test Dataset: Data collected from the specialized agents, where the true agent response is positive and other template responses are negative.

Key Insights and Ideas

  • Pretraining is Crucial: The most significant insight is that pretraining on a large, even noisy, unannotated dataset provides a substantial performance boost, far exceeding training from scratch. This suggests that general conversational understanding is a valuable foundation.
  • Ranking Loss Matters for Retrieval: For retrieval-based dialogue systems, a ranking-aware loss function (like the proposed Lranking) is more effective than standard classification losses (like BCE). This is because the ultimate goal is to rank the correct response highest, not just classify pairs as positive/negative.
  • Data-Mix Regularization Prevents Forgetting: Simply finetuning on the small, clean dataset can lead to overfitting and degradation of performance on the pretraining data. Data-mix regularization effectively mitigates this, allowing the model to retain its general knowledge while adapting to the specific task.
  • Real-World Deployment is Key: The paper emphasizes the importance of evaluating models in a live, human-in-the-loop setting. Their deployment in an agent-support application provides valuable insights into practical performance and failure modes.

Key Results

  • Offline Performance:
    • Pretraining with unlabeled data provided a 31% boost in Recall@1 compared to no pretraining.
  • Finetuning with a small annotated dataset resulted in 26% offline performance improvement in Recall@1 over the pretrained model.
  • Online Performance:
    • The finetuned model achieved 33% online performance improvement in Recall@1.
  • In the live customer service deployment, the model achieved 74% Recall@1.
  • The ranking loss finetuned model outperformed the cross-entropy loss finetuned model by 3% on MRR and 4% on Recall@1 on the finetune test dataset.
  • The ranking loss finetuned model achieved 74.0% Recall@1 on live customers, compared to 62.7% for BCE finetuning and 40.8% for the pretrained model alone.

Limitations and Future Work

  • Limitations:
    • Out-of-Domain Conversations: The model struggled when conversations deviated significantly from the common path seen in training data (17% of errors).
  • Missing Context: Agents sometimes had access to richer profile information than the model did, leading to errors (28% of errors related to extra profile features).
  • Subjectivity: 15% of rejections were due to agents preferring stylistically different but semantically similar responses, highlighting the subjective nature of dialogue.
  • Future Work:
    • Improved Contextual and Profile Features: The authors suggest investing in richer contextual and profile features to address the missing information issue.
  • Better Annotation and Evaluation: Further investment in standardizing agent training and evaluation methods is recommended to handle subjective preferences and improve overall model performance.
  • Generalizability: While the paper focuses on a delivery issue, the authors believe the pretrain-finetune approach can scale to other task-oriented dialogue systems.

Summary updated

BibTeX
@inproceedings{srivastava2021pretrain,
  author = {Manisha Srivastava and Yichao Lu and Riley Peschon and Chenyang Li},
  title = {{Pretrain-Finetune Based Training of Task-Oriented Dialogue Systems in a Real-World Setting}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-industry.5},
}
Figure