Abstract
This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks. We approach the problem from two aspects: supervised classifier probe and unsupervised mutual information probe. We fine-tune a feed-forward layer as the classifier probe on top of a fixed pre-trained language model with annotated labels in a supervised way. Meanwhile, we propose an unsupervised mutual information probe to evaluate the mutual dependence between a real clustering and a representation clustering. The goals of this empirical paper are to 1) investigate probing techniques, especially from the unsupervised mutual information aspect, 2) provide guidelines of pre-trained language model selection for the dialogue research community, 3) find insights of pre-training factors for dialogue application that may be the key to success.
Introduction
While Large Language Models (LLMs) like BERT and GPT-2 have revolutionized natural language processing, they are primarily trained on static, written text such as Wikipedia or books. Task-oriented dialogue (TOD), however, presents a unique challenge: it is conversational, goal-driven, and follows distinct linguistic patterns that general-purpose models may not fully capture.
This study evaluates 12 pre-trained language models to determine which ones provide the most "dialogue-aware" representations for task-oriented systems. By introducing a novel unsupervised mutual information probe alongside traditional supervised probes, we demonstrate that models pre-trained specifically on dialogue data—especially those using contrastive response selection objectives—significantly outperform general-purpose models.
Probing Methodology
To evaluate the intrinsic knowledge within frozen model representations, we employ two distinct probing strategies:
Supervised Classifier Probe
In this standard approach, the weights of the pre-trained model are frozen. A simple, shallow feed-forward layer is trained on top of the fixed embeddings to perform specific tasks like intent detection or domain identification. High performance from such a minimal layer implies that the underlying representation is highly informative and task-relevant.
Unsupervised Mutual Information Probe
To avoid the potential bias of a classifier learning to compensate for messy representations, we utilize clustering (K-means and Gaussian Mixture Models). We measure the Adjusted Normalized Mutual Information (ANMI) between these unsupervised clusters and ground-truth labels. This reveals how much task-relevant structure exists naturally in the embedding space without any task-specific training.
Experimental Setup
The study utilizes two major benchmarks:
- Multi-domain Wizard-of-Oz (MWOZ): A standard TOD dataset used for domain identification, dialogue act prediction, and slot tagging.
- Out-of-Scope (OOS) Intent Dataset: A massive intent detection dataset featuring 150 distinct intent classes plus an "out-of-scope" category.
Key Insights and Results
The Superiority of Dialogue Pre-training
Models trained on conversational data (like Reddit or specific TOD corpora) consistently outperformed general-purpose models. Specifically, models like ConveRT and TOD-BERT-jnt, which utilize a contrastive objective (learning to distinguish a correct response from a random one), produced significantly better representations than those using only Masked Language Modeling (MLM).
Task Difficulty and Representation Quality
Pre-trained models are naturally better at capturing high-level "gist" (intents and dialogue acts) than fine-grained details (slots like "price range" or "destination"). Slot information is poorly captured by frozen representations, often resulting in Micro-F1 scores below 30%.
The "Noisy Clustering" Phenomenon
Interestingly, models like GPT-2 and DialoGPT showed decent supervised classification accuracy but poor unsupervised clustering scores. This suggests their representations are "close" together in space (noisy) but still linearly separable—meaning they require more supervision to extract useful information compared to models like ConveRT.
Conclusion and Future Work
Our findings suggest that dialogue-specific pre-training is essential for building robust TOD systems. While probing fixed weights provides a clear picture of a model's inherent knowledge, future work should investigate if full fine-tuning can bridge the gap between general and specialized models. Additionally, developing pre-training objectives specifically designed to capture fine-grained slot information remains a critical area for improvement.
Summary updated