Abstract
Sequence-to-sequence models attempt to capture the correlation between all the words in the input and output sequences. While this is quite useful for machine translation where the correlation among the words is indeed quite strong, it becomes problematic for conversation modelling where the correlation is often at a much more abstract level. In contrast, humans tend to focus on the essential concepts discussed in the conversation context and generate responses accordingly. In this paper, we attempt to mimic this response-generating mechanism by learning the essential concepts in the context and response in an unsupervised manner. The proposed model, referred to as Mask & Focus, maps the input context to a sequence of concepts which are then used to generate the response concepts. Together, the context and the response concepts generate the final response. Mask & Focus achieves significant improvement over existing baselines in several established metrics for dialogues.
Introduction
Standard neural conversation models often suffer from a "goldfish memory" problem—they treat every word in a long dialogue history with equal weight. This lack of prioritization leads to the dreaded "generic response" syndrome, where a model defaults to safe but useless replies like "I don't know" or "I see." While word-to-word correlation works for machine translation, it often fails in dialogue. In a technical support chat, for example, the words "hello" and "yesterday" are far less important than "JDK7" or "password reset."
When models treat all words as equally important, the signal from critical keywords gets drowned out by the noise of common filler words. The authors argue that humans form a "rough sketch" of the important ideas (concepts) they want to convey before worrying about the exact phrasing.
Methodology: The Mask & Focus Framework
The "Mask & Focus" approach consists of two distinct phases designed to identify and prioritize these essential concepts.
1. Mask: Unsupervised Concept Discovery
To find important words without manual labels, the researchers use a technique called systematic probing. They take a pre-trained conversation model and "mask" (hide) words in the context one by one. They then calculate the Pointwise Mutual Information (PMI): if hiding a specific word causes the probability of the correct response to plummet, that word is deemed a "vital concept." This allows the model to build a "concept bank" entirely from the data without human intervention.
2. Focus: Concept-Guided Generation
Instead of going straight from context to response, the model follows a three-step generation process:
- Context Encoding: The model encodes the utterances along with the identified context concepts.
- Concept Prediction: The model first predicts a set of response concepts (a semantic "sketch").
- Final Response: A decoder uses a copy mechanism to take those predicted concepts and "flesh them out" into a natural language sentence.
To ensure the model pays attention to these concepts, the authors optimize a Variational Lower Bound (ELBO) objective, specifically weighting the training to favor words with high mutual information.
Experimental Results
The model was evaluated on the Ubuntu Dialogue Corpus v2.0 and an internal Tech Support dataset. The results demonstrated that the "concept bridge" significantly reduces genericity by forcing the model to predict response concepts before the full sentence.
Key performance metrics include:
- Entity F1 Score: On the Ubuntu corpus, Mask & Focus achieved a score of 7.82, more than triple the score of VHRED (2.53), indicating much higher accuracy in mentioning correct technical terms.
- Human Evaluation: In a blind study, humans rated Mask & Focus responses with a Relevance score of 1.79 (on a 0-4 scale), significantly higher than HRED (1.25).
- Generic Response Ratio: The ratio dropped to 0.18, compared to 0.53 for HRED, meaning the model is nearly three times less likely to give a "canned" response.
Conclusion
Mask & Focus demonstrates that unsupervised discovery of concepts can outperform manual labels, identifying technical entities that human annotators might miss. By mimicking human intuition and focusing on the "crux" of a conversation, the model generates significantly more informative and topically relevant replies. Future work will look into automating PMI thresholds and handling abstract, latent concepts that are understood but not explicitly mentioned in the text.
Summary updated