Abstract
Sequence-to-sequence models attempt to capture the correlation between all words in input and output sequences. While effective for machine translation, this approach is problematic for conversation modelling where correlations exist at a more abstract level. Humans focus on essential concepts in a conversation context to generate responses. We propose Mask & Focus, a model that mimics this mechanism by learning essential concepts in an unsupervised manner. The model maps input context to a sequence of concepts, which are then used to generate response concepts and the final response. By masking words in the input and observing the effect on response generation, the model learns context concepts and focuses on response concepts with high mutual information. Mask & Focus achieves significant improvements over existing baselines across several dialogue metrics.
Introduction
Standard neural conversation models, such as the Hierarchical Recurrent Encoder-Decoder (HRED), typically follow a sequence-to-sequence (seq2seq) approach. While this architecture excels in machine translation—where source and target words often have direct correspondences—it frequently falters in open-ended dialogue. In a conversation, not all words carry equal weight. For instance, in the query "How do I reset my password on Ubuntu?", "password" and "reset" are vital concepts, whereas "How," "do," and "I" are functional fillers.
Standard models treat every word with equal importance, often leading them to generate generic, high-probability responses like "I don't know" or "I see." Mask & Focus addresses this by identifying core concepts within a context and using them as anchors for response generation, mirroring human cognitive processes.
The Mask & Focus Framework
The Mask & Focus approach operates in two distinct phases:
- Mask (Concept Discovery): To identify "concepts," the framework probes a pre-trained conversation model. It systematically "masks" (hides) words in the input context and measures the Pointwise Mutual Information (PMI) between the masked word and the actual response. If hiding a word significantly decreases the likelihood of the correct response, that word is identified as a "vital concept."
- Focus (Response Generation): Rather than mapping context directly to a response, the model introduces an intermediate conceptual step. It encodes the identified context concepts and predicts a set of "response concepts"—a skeletal outline of the intended answer. Finally, a decoder utilizes both the original context and these predicted concepts to generate a natural language response, often employing a copy mechanism to ensure specific technical terms are preserved.
The model is trained by optimizing an Evidence Lower Bound (ELBO), with a variational distribution designed to prioritize words with high mutual information.
Experimental Evaluation
The authors evaluated Mask & Focus on two challenging technical domains:
- Ubuntu Dialogue Corpus v2.0: A large-scale dataset of approximately 500,000 training pairs from Ubuntu chat logs.
- Internal Tech Support Dataset: A smaller, focused dataset (~20,000 conversations) used to test performance in low-resource settings.
Key Results and Insights
The model was benchmarked against baselines like HRED and VHRED using metrics tailored for technical dialogue:
- Technical Accuracy: On the Ubuntu dataset, Mask & Focus achieved an Entity F1 score of 7.82, nearly tripling the performance of standard HRED (2.22).
- Human Evaluation: In a blind study, human raters scored the relevance of Mask & Focus responses at 1.79/4.0, significantly higher than HRED’s 1.25/4.0.
- Reducing Generic Responses: Mask & Focus dramatically reduced the Generic Response (GR) Ratio. While HRED produced generic "filler" responses 53% of the time, Mask & Focus reduced this to just 18%.
Key Insights
- Unsupervised Advantage: Mask & Focus outperformed supervised models (like MrRNN-ActEnt) that relied on predefined entity lists. This suggests that "probing" a model to discover what it finds important is more effective than manual labeling.
- Semantic-Syntactic Disentanglement: By predicting response concepts first, the model separates "what to say" (semantics) from "how to say it" (language modeling).
- Data Efficiency: In low-resource experiments (5k–10k samples), Mask & Focus maintained significantly higher BLEU scores than HRED, demonstrating that learning concepts is more data-efficient than learning word-level correlations from scratch.
Conclusion and Future Work
Mask & Focus represents a significant advancement in generating relevant, topically coherent dialogue. However, some limitations remain. Currently, context concepts are restricted to words present in the input; the model cannot yet generate abstract concepts not explicitly mentioned. Additionally, the "Mask" phase depends on the quality of a pre-trained generative model.
Future research may extend this framework to incorporate external knowledge bases, allowing the model to "focus" on concepts retrieved from technical documentation or FAQs, further enhancing its utility in specialized domains.
Summary updated