Improving Matching Models with Hierarchical Contextualized Representations for Multi-turn Response Selection

Chongyang Tao, Wei Wu, Yansong Feng, Dongyan Zhao, Rui YanSIGIR 2020 · DOI · OpenReview

Introduction

Retrieval-based chatbots aim to select the most appropriate response from a candidate pool given a multi-turn conversation history. A critical component of this process is the representation of words and sentences. While traditional word embeddings like Word2Vec or GloVe provide a foundation, they fail to capture the nuances of context. Recent advances in contextualized embeddings, such as ELMo and CoVe, have shown promise but are often trained on formal, non-hierarchical text, making them less effective for the informal and structured nature of human dialogue.

The Hierarchical Challenge in Dialogue

Conversations possess a unique hierarchical structure: words form utterances, and sequences of utterances form a session. General-purpose contextualized models typically treat input as flat sequences, overlooking the global session-level flow. Furthermore, there is a significant data discrepancy; models trained on Wikipedia or news articles struggle with the informal language, domain-specific jargon (e.g., technical support terms), and unique structural patterns of chat data.

Methodology: The ECMo Framework

To address these gaps, the authors propose ECMo (Embeddings from a COnversation MOdel), derived from a Hierarchical Encoder-Decoder (HED) architecture pre-trained on massive, unlabeled conversation datasets.

Hierarchical Encoder-Decoder (HED) Architecture

The HED model consists of two primary encoding layers designed to capture multi-level context:

  1. Utterance-Level Encoder: A bidirectional GRU (biGRU) with max-pooling processes individual sentences to capture local context—the relationships between words within a single turn.
  2. Context-Level Encoder: A second GRU layer takes the summarized outputs of the utterance-level encoder as input, capturing the global context and the progression of the conversation across multiple turns.
The Hierarchical Encoder-Decoder (HED) architecture used for pre-training ECMo.

During pre-training, a decoder is used to predict the next utterance, forcing the encoders to learn rich, representative features of the dialogue history. Once trained, these encoders generate two types of embeddings: ECMo_local (word-level) for the input layer of matching models and ECMo_global (sentence-level) to help calculate final matching scores.

Experimental Evaluation

The effectiveness of ECMo was tested on two benchmark datasets: the Ubuntu Dialogue Corpus (technical support) and the Douban Conversation Corpus (social networking). Pre-training was conducted on even larger datasets: 10 million Twitter conversations for English and 5 million Sina Weibo conversations for Chinese.

Key Results

Integrating ECMo into state-of-the-art matching models like Multi-view and SMN yielded significant performance improvements.

Performance comparison of ECMo against baseline models and other contextualized embeddings on Ubuntu and Douban corpora.

As shown in the results table, ECMo consistently outperformed general-purpose embeddings like ELMo and CoVe. On the Ubuntu dataset, ECMo improved the $R_{10}@1$ metric by 4.3% for the Multi-view model and 2.4% for SMN. Notably, ECMo surpassed even fine-tuned versions of ELMo, highlighting the importance of domain-specific pre-training on actual dialogue data.

Discussion and Insights

The success of ECMo underscores two vital principles in dialogue modeling:

  • Hierarchy is Essential: Modeling both local (sentence-level) and global (session-level) context is crucial for accurate response selection. Ablation studies confirmed that removing either component leads to a performance decline.
  • Domain Alignment: Training on informal, "in-the-wild" dialogue (Twitter/Weibo) provides a superior foundation for chatbots compared to massive models trained on formal prose.

Conclusion

ECMo demonstrates that hierarchical, domain-specific pre-training is a powerful strategy for improving multi-turn response selection. By capturing the unique structural and linguistic properties of human conversation, ECMo provides a more robust representation than general-purpose contextualized embeddings, setting a new standard for retrieval-based chatbot components.

Summary updated

Abstract

In this paper, we study context-response matching with pre-trained contextualized representations for multi-turn response selection in retrieval-based chatbots. Existing models, such as Cove and ELMo, are trained with limited context (often a single sentence or paragraph), and may not work well on multi-turn conversations, due to the hierarchical nature, informal language, and domain-specific words. To address the challenges, we propose pre-training hierarchical contextualized representations, including contextual word-level and sentence-level representations, by learning a dialogue generation model from large-scale conversations with a hierarchical encoder-decoder architecture. Then the two levels of representations are blended into the input and output layer of a matching model respectively. Experimental results on two benchmark conversation datasets indicate that the proposed hierarchical contextualized representations can bring significantly and consistently improvement to existing matching models for response selection.
BibTeX
@inproceedings{tao2020improving,
  author = {Chongyang Tao and Wei Wu and Yansong Feng and Dongyan Zhao and Rui Yan},
  title = {{Improving Matching Models with Hierarchical Contextualized Representations for Multi-turn Response Selection}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  doi = {10.1145/3397271.3401290},
}
Figure