Multi-Granularity Representations of Dialog

Shikib Mehri, Maxine EskénaziEMNLP 2019 · DOI

TL;DR

This paper introduces Multi-Granularity Training (MGT), a novel method to train dialog models that learn representations at multiple levels of detail. By carefully sampling negative examples during training, MGT encourages models to capture both fine-grained word-level nuances and broader abstract dialog properties, leading to improved performance on next utterance retrieval and better generalization.

What Problem Does This Paper Address?

Natural language processing models, especially in dialog systems, rely on learning effective "latent representations" of language – essentially, compact numerical summaries that capture the meaning of text. For dialog, these representations need to be rich enough to track user goals, entities, co-references, and the overall communicative purpose of an utterance. While large-scale pre-training on general text has shown promise, these representations often struggle when applied to dialogs because dialog data has different characteristics (domain, style) and pre-training objectives don't always capture the multi-faceted nature of dialog. Fine-tuning these large models for dialog is also computationally expensive. This paper aims to address this by developing a training procedure that explicitly learns diverse representations tailored for dialog tasks, without requiring extensive fine-tuning of pre-trained models.

Methods

The core innovation is Multi-Granularity Training (MGT). Instead of randomly sampling negative responses (incorrect answers) during training, MGT uses a controlled sampling strategy. The idea is that different types of negative examples encourage different kinds of learning:

  • Lexically similar negative examples (e.g., responses that use similar words but are still incorrect) push the model to learn fine-grained, word-level representations.
  • Semantically distant negative examples (e.g., responses that are completely off-topic) push the model to learn more abstract, high-level representations of the dialog.

MGT achieves this by:

  1. Calculating Semantic Similarity: Using the cosine similarity between response embeddings (generated by a response encoder) to measure how semantically close two utterances are.
  2. Segmenting Negative Candidates: For a given dialog context and ground-truth response, MGT divides all other responses in the training set into L segments based on their semantic distance to the ground-truth response.
  3. Training Multiple Models: L separate models are trained, each using negative candidates sampled exclusively from one of these segments. This ensures each model learns representations at a specific level of granularity.
  4. Ensembling: The outputs of these L models are then ensembled to produce the final prediction.

MGT is designed to be architecture-agnostic, meaning it can be applied to various neural network architectures for dialog.

Datasets

The paper evaluates MGT on two widely used dialog datasets:

  • MultiWOZ: A large-scale, multi-domain dataset for task-oriented dialog. It was converted into a retrieval corpus with 8,422 training dialogs, 1,000 validation, and 1,000 test dialogs, each with 20 candidate responses.
  • Ubuntu Dialog Corpus: A large corpus of informal, task-oriented conversations from IRC, used for next utterance retrieval. The training set was modified for retrieval, resulting in 499,873 examples, with 10 candidate responses per dialog.

Key Insights and Ideas

  • Diversity through Controlled Sampling: The key insight is that the type of negative examples used during training directly influences the type of representations learned. MGT leverages this by explicitly controlling the semantic distance of negative samples.
  • Granularity Matters for Dialog: Dialog is complex and requires understanding at multiple levels. MGT's success suggests that explicitly training models to capture these different granularities (from specific words to overall user goals) is crucial for robust dialog understanding.
  • MGT as a General Training Paradigm: The method is presented as a general training procedure that can enhance existing architectures, rather than a new architecture itself. This makes it broadly applicable.
  • Improved Transferability: MGT-trained models show better performance on downstream tasks (like bag-of-words prediction and dialog act prediction) even without fine-tuning, indicating they learn more generalizable representations.

Key Results

  • Next Utterance Retrieval Performance: MGT consistently outperformed baseline dual encoder models and standard ensembles of dual encoders on both MultiWOZ and Ubuntu datasets, achieving statistically significant improvements in Mean Reciprocal Rank (MRR) and Hits@1. For example, on MultiWOZ, MGT achieved an MRR of 82.74% compared to 81.53% for a similarly sized ensemble.
  • Explicit Granularity Modeling: Experiments using frozen MGT models on downstream tasks confirmed that models trained with higher granularity samples excelled at word-level prediction (bag-of-words), while models trained with lower granularity samples were better at abstract dialog act prediction.
  • Generalizability: MGT models showed improved performance on downstream tasks without fine-tuning, indicating better transferability compared to standard models and ensembles. For instance, on the dialog act prediction task, MGT achieved 22.85% (DA F-1) compared to 22.39% for an ensemble, and significantly outperformed the baseline dual encoder (19.09%).

Limitations and Future Work

  • Semantic Similarity Measure: The paper uses cosine similarity as a measure of semantic distance. While robust to noise, the authors suggest future work could explore whether a better distance measure would further improve MGT.
  • Combining Granularities: The current method ensembles the outputs of models trained at different granularities. A more sophisticated mechanism for combining these representations could be explored.
  • Generalizing MGT: The authors propose extending MGT to other tasks beyond next utterance retrieval, such as language generation. They also suggest generalizing it to learn representations along other axes, like domain, style, or intent.

Summary updated

BibTeX
@inproceedings{mehri2019multi,
  author = {Shikib Mehri and Maxine Eskénazi},
  title = {{Multi-Granularity Representations of Dialog}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1184},
}
Figure