Mask & Focus: Conversation Modelling by Learning Concepts

Gaurav Pandey, Dinesh Raghu, Sachindra Joshi

Abstract

Sequence-to-sequence models attempt to capture the correlation between all the words in the input and output sequences. While this is quite useful for machine translation where the correlation among the words is indeed quite strong, it becomes problematic for conversation modelling where the correlation is often at a much more abstract level. In contrast, humans tend to focus on the essential concepts discussed in the conversation context and generate responses accordingly. In this paper, we attempt to mimic this response-generating mechanism by learning the essential concepts in the context and response in an unsupervised manner. The proposed model, referred to as Mask & Focus, maps the input context to a sequence of concepts which are then used to generate the response concepts. Together, the context and the response concepts generate the final response. Mask & Focus achieves significant improvement over existing baselines in several established metrics for dialogues.

Introduction

Standard neural conversation models often suffer from a "goldfish memory" problem—they treat every word in a long dialogue history with equal weight. This lack of prioritization leads to the dreaded "generic response" syndrome, where a model defaults to safe but useless replies like "I don't know" or "I see." While word-to-word correlation works for machine translation, it often fails in dialogue. In a technical support chat, for example, the words "hello" and "yesterday" are far less important than "JDK7" or "password reset."

When models treat all words as equally important, the signal from critical keywords gets drowned out by the noise of common filler words. The authors argue that humans form a "rough sketch" of the important ideas (concepts) they want to convey before worrying about the exact phrasing.

Methodology: The Mask & Focus Framework

The "Mask & Focus" approach consists of two distinct phases designed to identify and prioritize these essential concepts.

1. Mask: Unsupervised Concept Discovery

To find important words without manual labels, the researchers use a technique called systematic probing. They take a pre-trained conversation model and "mask" (hide) words in the context one by one. They then calculate the Pointwise Mutual Information (PMI): if hiding a specific word causes the probability of the correct response to plummet, that word is deemed a "vital concept." This allows the model to build a "concept bank" entirely from the data without human intervention.

2. Focus: Concept-Guided Generation

Instead of going straight from context to response, the model follows a three-step generation process:

  1. Context Encoding: The model encodes the utterances along with the identified context concepts.
  2. Concept Prediction: The model first predicts a set of response concepts (a semantic "sketch").
  3. Final Response: A decoder uses a copy mechanism to take those predicted concepts and "flesh them out" into a natural language sentence.
The Mask & Focus architecture uses a concept-guided generation process to bridge context and response.

To ensure the model pays attention to these concepts, the authors optimize a Variational Lower Bound (ELBO) objective, specifically weighting the training to favor words with high mutual information.

Experimental Results

The model was evaluated on the Ubuntu Dialogue Corpus v2.0 and an internal Tech Support dataset. The results demonstrated that the "concept bridge" significantly reduces genericity by forcing the model to predict response concepts before the full sentence.

A comparison of dialogue responses shows how Mask & Focus identifies key concepts to generate more relevant replies than standard models.

Key performance metrics include:

  • Entity F1 Score: On the Ubuntu corpus, Mask & Focus achieved a score of 7.82, more than triple the score of VHRED (2.53), indicating much higher accuracy in mentioning correct technical terms.
  • Human Evaluation: In a blind study, humans rated Mask & Focus responses with a Relevance score of 1.79 (on a 0-4 scale), significantly higher than HRED (1.25).
  • Generic Response Ratio: The ratio dropped to 0.18, compared to 0.53 for HRED, meaning the model is nearly three times less likely to give a "canned" response.

Conclusion

Mask & Focus demonstrates that unsupervised discovery of concepts can outperform manual labels, identifying technical entities that human annotators might miss. By mimicking human intuition and focusing on the "crux" of a conversation, the model generates significantly more informative and topically relevant replies. Future work will look into automating PMI thresholds and handling abstract, latent concepts that are understood but not explicitly mentioned in the text.

Summary updated

Abstract

Sequence to sequence models attempt to capture the correlation between all the words in the input and output sequences. While this is quite useful for machine translation where the correlation among the words is indeed quite strong, it becomes problematic for conversation modelling where the correlation is often at a much abstract level. In contrast, humans tend to focus on the essential concepts discussed in the conversation context and generate responses accordingly. In this paper, we attempt to mimic this response generating mechanism by learning the essential concepts in the context and response in an unsupervised manner. The proposed model, referred to as Mask \& Focus maps the input context to a sequence of concepts which are then used to generate the response concepts. Together, the context and the response concepts generate the final response. In order to learn context concepts from the training data automatically, we \emph{mask} words in the input and observe the effect of masking on response generation. We train our model to learn those response concepts that have high mutual information with respect to the context concepts, thereby guiding the model to \emph{focus} on the context concepts. Mask \& Focus achieves significant improvement over the existing baselines in several established metrics for dialogues.
BibTeX
@inproceedings{pandey2020mask,
  author = {Gaurav Pandey and Dinesh Raghu and Sachindra Joshi},
  title = {{Mask \& Focus: Conversation Modelling by Learning Concepts}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=_hYVHoe30pI},
}
Figure