Mask & Focus: Conversation Modelling by Learning Concepts

Gaurav Pandey, Dinesh Raghu, Sachindra Joshi

Abstract

Sequence-to-sequence models attempt to capture the correlation between all words in input and output sequences. While effective for machine translation, this approach is problematic for conversation modelling where correlations exist at a more abstract level. Humans focus on essential concepts in a conversation context to generate responses. We propose Mask & Focus, a model that mimics this mechanism by learning essential concepts in an unsupervised manner. The model maps input context to a sequence of concepts, which are then used to generate response concepts and the final response. By masking words in the input and observing the effect on response generation, the model learns context concepts and focuses on response concepts with high mutual information. Mask & Focus achieves significant improvements over existing baselines across several dialogue metrics.

Introduction

Standard neural conversation models, such as the Hierarchical Recurrent Encoder-Decoder (HRED), typically follow a sequence-to-sequence (seq2seq) approach. While this architecture excels in machine translation—where source and target words often have direct correspondences—it frequently falters in open-ended dialogue. In a conversation, not all words carry equal weight. For instance, in the query "How do I reset my password on Ubuntu?", "password" and "reset" are vital concepts, whereas "How," "do," and "I" are functional fillers.

Standard models treat every word with equal importance, often leading them to generate generic, high-probability responses like "I don't know" or "I see." Mask & Focus addresses this by identifying core concepts within a context and using them as anchors for response generation, mirroring human cognitive processes.

The Mask & Focus Framework

The Mask & Focus approach operates in two distinct phases:

  1. Mask (Concept Discovery): To identify "concepts," the framework probes a pre-trained conversation model. It systematically "masks" (hides) words in the input context and measures the Pointwise Mutual Information (PMI) between the masked word and the actual response. If hiding a word significantly decreases the likelihood of the correct response, that word is identified as a "vital concept."
  2. Focus (Response Generation): Rather than mapping context directly to a response, the model introduces an intermediate conceptual step. It encodes the identified context concepts and predicts a set of "response concepts"—a skeletal outline of the intended answer. Finally, a decoder utilizes both the original context and these predicted concepts to generate a natural language response, often employing a copy mechanism to ensure specific technical terms are preserved.
Comparison of response generation between standard HRED and the Mask & Focus model, highlighting identified key concepts.

The model is trained by optimizing an Evidence Lower Bound (ELBO), with a variational distribution designed to prioritize words with high mutual information.

Experimental Evaluation

The authors evaluated Mask & Focus on two challenging technical domains:

  • Ubuntu Dialogue Corpus v2.0: A large-scale dataset of approximately 500,000 training pairs from Ubuntu chat logs.
  • Internal Tech Support Dataset: A smaller, focused dataset (~20,000 conversations) used to test performance in low-resource settings.

Key Results and Insights

The model was benchmarked against baselines like HRED and VHRED using metrics tailored for technical dialogue:

  • Technical Accuracy: On the Ubuntu dataset, Mask & Focus achieved an Entity F1 score of 7.82, nearly tripling the performance of standard HRED (2.22).
  • Human Evaluation: In a blind study, human raters scored the relevance of Mask & Focus responses at 1.79/4.0, significantly higher than HRED’s 1.25/4.0.
  • Reducing Generic Responses: Mask & Focus dramatically reduced the Generic Response (GR) Ratio. While HRED produced generic "filler" responses 53% of the time, Mask & Focus reduced this to just 18%.

Key Insights

  • Unsupervised Advantage: Mask & Focus outperformed supervised models (like MrRNN-ActEnt) that relied on predefined entity lists. This suggests that "probing" a model to discover what it finds important is more effective than manual labeling.
  • Semantic-Syntactic Disentanglement: By predicting response concepts first, the model separates "what to say" (semantics) from "how to say it" (language modeling).
  • Data Efficiency: In low-resource experiments (5k–10k samples), Mask & Focus maintained significantly higher BLEU scores than HRED, demonstrating that learning concepts is more data-efficient than learning word-level correlations from scratch.

Conclusion and Future Work

Mask & Focus represents a significant advancement in generating relevant, topically coherent dialogue. However, some limitations remain. Currently, context concepts are restricted to words present in the input; the model cannot yet generate abstract concepts not explicitly mentioned. Additionally, the "Mask" phase depends on the quality of a pre-trained generative model.

Future research may extend this framework to incorporate external knowledge bases, allowing the model to "focus" on concepts retrieved from technical documentation or FAQs, further enhancing its utility in specialized domains.

Summary updated

Abstract

Sequence to sequence models attempt to capture the correlation between all the words in the input and output sequences. While this is quite useful for machine translation where the correlation among the words is indeed quite strong, it becomes problematic for conversation modelling where the correlation is often at a much abstract level. In contrast, humans tend to focus on the essential concepts discussed in the conversation context and generate responses accordingly. In this paper, we attempt to mimic this response generating mechanism by learning the essential concepts in the context and response in an unsupervised manner. The proposed model, referred to as Mask & Focus maps the input context to a sequence of concepts which are then used to generate the response concepts. Together, the context and the response concepts generate the final response. In order to learn context concepts from the training data automatically, we mask words in the input and observe the effect of masking on response generation. We train our model to learn those response concepts that have high mutual information with respect to the context concepts, thereby guiding the model to focus on the context concepts. Mask & Focus achieves significant improvement over the existing baselines in several established metrics for dialogues.
BibTeX
@inproceedings{pandey2020mask,
  author = {Gaurav Pandey and Dinesh Raghu and Sachindra Joshi},
  title = {{Mask \& Focus: Conversation Modelling by Learning Concepts}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=_hYVHoe30pI},
}
Figure