Abstract
Chien-Sheng Wu, Caiming Xiong. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
While general-purpose language models (LMs) like BERT and GPT-2 have revolutionized natural language processing, a significant distribution gap exists between the formal text they are trained on (e.g., Wikipedia, books) and the goal-directed, multi-turn nature of human conversation. In task-oriented dialogue (TOD), models must track specific intents, slots (e.g., "price range"), and system actions. This study evaluates 12 pre-trained LMs to determine which models intrinsically "understand" dialogue best and whether this understanding stems from training data or specific pre-training objectives.
Probing Methodologies
The researchers utilize two distinct "probing" techniques to analyze the frozen representations of the 12 LMs:
- Supervised Classifier Probe: A simple, single-layer feed-forward network is trained on top of the frozen LM for specific tasks like intent detection. Success indicates that the frozen representation already contains the necessary information.
- Unsupervised Mutual Information (MI) Probe: Raw embeddings are clustered using K-means or Gaussian Mixture Models (GMM) without labels. The Adjusted Normalized Mutual Information (ANMI) between these clusters and ground-truth labels is measured to reveal how "naturally" the model organizes dialogue concepts in its vector space.
Datasets
The evaluation leverages several cornerstone dialogue benchmarks:
- Multi-domain Wizard-of-Oz (MWOZ): Used for domain identification, dialogue slot tagging, and dialogue act prediction.
- Out-of-Scope (OOS) Intent Dataset: A large-scale dataset with 150 intent classes plus "out-of-scope" queries for testing fine-grained intent detection.
- Dialogue-Specific Pre-training Data: Reference datasets include the 727M Reddit pairs used for ConveRT and the nine TOD datasets aggregated for TOD-BERT.
Results and Discussion
The Impact of Dialogue-Specific Pre-training
The results demonstrate that models trained on conversational data (ConveRT, TOD-BERT, DialoGPT) consistently outperform those trained on general text (BERT, RoBERTa), regardless of model size. As shown in the supervised probing results, dialogue-specific models excel across various tasks including domain and intent detection.
Contrastive Learning and the GPT Paradox
Contrastive learning objectives, such as "response selection" used in TOD-BERT-jnt and ConveRT, create much tighter and more meaningful clusters than Masked Language Modeling (MLM) alone. However, a "GPT Paradox" was observed: while GPT-2 and DialoGPT perform well on supervised probes, they fail significantly on unsupervised clustering. Their internal representations appear "noisier" or less linearly separable than BERT-based models, suggesting they are better at generating text than organizing semantic concepts.
Intent vs. Slot Representation
All models are significantly better at capturing high-level intent and dialogue acts than fine-grained "slots." Slot information appears poorly captured in global [CLS] or mean-pooled representations, as evidenced by the relatively low performance in slot tagging tasks compared to intent detection.
Conclusion
This study highlights that pre-training on dialogue-specific data and employing contrastive learning are the primary drivers for high-quality dialogue representations. Future research should explore layer-wise probing to understand how dialogue information is constructed through Transformer blocks and develop more localized probing techniques to better capture token-level slot information.
Summary updated