Probing Task-Oriented Dialogue Representation from Language Models

Chien-Sheng Wu, Caiming Xiong

Abstract

Chien-Sheng Wu, Caiming Xiong. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

While general-purpose language models (LMs) like BERT and GPT-2 have revolutionized natural language processing, a significant distribution gap exists between the formal text they are trained on (e.g., Wikipedia, books) and the goal-directed, multi-turn nature of human conversation. In task-oriented dialogue (TOD), models must track specific intents, slots (e.g., "price range"), and system actions. This study evaluates 12 pre-trained LMs to determine which models intrinsically "understand" dialogue best and whether this understanding stems from training data or specific pre-training objectives.

Probing Methodologies

The researchers utilize two distinct "probing" techniques to analyze the frozen representations of the 12 LMs:

  1. Supervised Classifier Probe: A simple, single-layer feed-forward network is trained on top of the frozen LM for specific tasks like intent detection. Success indicates that the frozen representation already contains the necessary information.
  2. Unsupervised Mutual Information (MI) Probe: Raw embeddings are clustered using K-means or Gaussian Mixture Models (GMM) without labels. The Adjusted Normalized Mutual Information (ANMI) between these clusters and ground-truth labels is measured to reveal how "naturally" the model organizes dialogue concepts in its vector space.

Datasets

The evaluation leverages several cornerstone dialogue benchmarks:

  • Multi-domain Wizard-of-Oz (MWOZ): Used for domain identification, dialogue slot tagging, and dialogue act prediction.
  • Out-of-Scope (OOS) Intent Dataset: A large-scale dataset with 150 intent classes plus "out-of-scope" queries for testing fine-grained intent detection.
  • Dialogue-Specific Pre-training Data: Reference datasets include the 727M Reddit pairs used for ConveRT and the nine TOD datasets aggregated for TOD-BERT.

Results and Discussion

The Impact of Dialogue-Specific Pre-training

The results demonstrate that models trained on conversational data (ConveRT, TOD-BERT, DialoGPT) consistently outperform those trained on general text (BERT, RoBERTa), regardless of model size. As shown in the supervised probing results, dialogue-specific models excel across various tasks including domain and intent detection.

Supervised probing results across four TOD tasks: (a) MWOZ Domain, (b) OOS Intent, (c) MWOZ Slot, and (d) MWOZ Act.

Contrastive Learning and the GPT Paradox

Contrastive learning objectives, such as "response selection" used in TOD-BERT-jnt and ConveRT, create much tighter and more meaningful clusters than Masked Language Modeling (MLM) alone. However, a "GPT Paradox" was observed: while GPT-2 and DialoGPT perform well on supervised probes, they fail significantly on unsupervised clustering. Their internal representations appear "noisier" or less linearly separable than BERT-based models, suggesting they are better at generating text than organizing semantic concepts.

Unsupervised probing results (ANMI scores) for MWOZ Slot-Set across different cluster counts, illustrating the natural organization of dialogue concepts.

Intent vs. Slot Representation

All models are significantly better at capturing high-level intent and dialogue acts than fine-grained "slots." Slot information appears poorly captured in global [CLS] or mean-pooled representations, as evidenced by the relatively low performance in slot tagging tasks compared to intent detection.

Conclusion

This study highlights that pre-training on dialogue-specific data and employing contrastive learning are the primary drivers for high-quality dialogue representations. Future research should explore layer-wise probing to understand how dialogue information is constructed through Transformer blocks and develop more localized probing techniques to better capture token-level slot information.

Summary updated

BibTeX
@inproceedings{wu2020probing,
  author = {Chien-Sheng Wu and Caiming Xiong},
  title = {{Probing Task-Oriented Dialogue Representation from Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=bbPqhGFK8lm},
}
Figure