Introduction
Retrieval-based chatbots aim to select the most appropriate response from a candidate pool given a multi-turn conversation history. A critical component of this process is the representation of words and sentences. While traditional word embeddings like Word2Vec or GloVe provide a foundation, they fail to capture the nuances of context. Recent advances in contextualized embeddings, such as ELMo and CoVe, have shown promise but are often trained on formal, non-hierarchical text, making them less effective for the informal and structured nature of human dialogue.
The Hierarchical Challenge in Dialogue
Conversations possess a unique hierarchical structure: words form utterances, and sequences of utterances form a session. General-purpose contextualized models typically treat input as flat sequences, overlooking the global session-level flow. Furthermore, there is a significant data discrepancy; models trained on Wikipedia or news articles struggle with the informal language, domain-specific jargon (e.g., technical support terms), and unique structural patterns of chat data.
Methodology: The ECMo Framework
To address these gaps, the authors propose ECMo (Embeddings from a COnversation MOdel), derived from a Hierarchical Encoder-Decoder (HED) architecture pre-trained on massive, unlabeled conversation datasets.
Hierarchical Encoder-Decoder (HED) Architecture
The HED model consists of two primary encoding layers designed to capture multi-level context:
- Utterance-Level Encoder: A bidirectional GRU (biGRU) with max-pooling processes individual sentences to capture local context—the relationships between words within a single turn.
- Context-Level Encoder: A second GRU layer takes the summarized outputs of the utterance-level encoder as input, capturing the global context and the progression of the conversation across multiple turns.
During pre-training, a decoder is used to predict the next utterance, forcing the encoders to learn rich, representative features of the dialogue history. Once trained, these encoders generate two types of embeddings: ECMo_local (word-level) for the input layer of matching models and ECMo_global (sentence-level) to help calculate final matching scores.
Experimental Evaluation
The effectiveness of ECMo was tested on two benchmark datasets: the Ubuntu Dialogue Corpus (technical support) and the Douban Conversation Corpus (social networking). Pre-training was conducted on even larger datasets: 10 million Twitter conversations for English and 5 million Sina Weibo conversations for Chinese.
Key Results
Integrating ECMo into state-of-the-art matching models like Multi-view and SMN yielded significant performance improvements.
As shown in the results table, ECMo consistently outperformed general-purpose embeddings like ELMo and CoVe. On the Ubuntu dataset, ECMo improved the $R_{10}@1$ metric by 4.3% for the Multi-view model and 2.4% for SMN. Notably, ECMo surpassed even fine-tuned versions of ELMo, highlighting the importance of domain-specific pre-training on actual dialogue data.
Discussion and Insights
The success of ECMo underscores two vital principles in dialogue modeling:
- Hierarchy is Essential: Modeling both local (sentence-level) and global (session-level) context is crucial for accurate response selection. Ablation studies confirmed that removing either component leads to a performance decline.
- Domain Alignment: Training on informal, "in-the-wild" dialogue (Twitter/Weibo) provides a superior foundation for chatbots compared to massive models trained on formal prose.
Conclusion
ECMo demonstrates that hierarchical, domain-specific pre-training is a powerful strategy for improving multi-turn response selection. By capturing the unique structural and linguistic properties of human conversation, ECMo provides a more robust representation than general-purpose contextualized embeddings, setting a new standard for retrieval-based chatbot components.
Summary updated