Abstract
Generating informative and contextually relevant responses remains a significant challenge for open-domain chatbots. Standard Sequence-to-Sequence (S2S) models often default to generic "safe" replies like "I don't know." This paper presents S2SPMN (Sequence to Sequence with Prototype Memory Network), a framework that retrieves relevant "prototype" responses from a training corpus to guide the generation process. By employing a hierarchical attention mechanism, S2SPMN effectively integrates these prototypes, resulting in responses that are significantly more natural and relevant than those produced by baseline models.
Introduction
Response Generation (RG) in chatbots is often framed similarly to Machine Translation (MT), where a post is mapped to a response. However, unlike MT, which preserves meaning across languages, RG requires a "meaning transformation" from a prompt to a logically following reply. Standard neural models frequently fail this transformation, falling into a local optimum of generating high-frequency, generic phrases. The authors propose that the rich contextual information within a model's own training data—specifically, previous responses to similar prompts—can serve as "soft prototypes" to guide the generation of more specific and content-rich content.
The S2SPMN Framework
The S2SPMN architecture enhances the standard LSTM-based encoder-decoder model with a Prototype Memory Network (PMN) and a hierarchical attention mechanism.
The framework consists of three core components:
- Base Model: A standard encoder-decoder architecture with a traditional attention mechanism.
- Prototype Memory Network (PMN): This component stores a set of "prototypes" (responses from the training set). The authors evaluated two variants:
- Static PMN (SPMN): Uses a fixed set of randomly selected responses.
- Dynamic PMN (DPMN): For each input post, the model retrieves the most relevant responses from the training set using TF-IDF and cosine similarity.
- Hierarchical Attention Mechanism: This two-stage process allows the model to synthesize information from multiple prototypes. It first performs sentence-level attention to create an "abstractive prototype" vector and then word-level attention to assist the decoder in word selection.
Experimental Setup
The model was evaluated using a subset of the STC (Short Text Conversation) dataset, derived from Weibo. The training set comprised approximately 316,000 post-response pairs. To ensure computational efficiency, sentences were limited to 25 words, and the vocabulary was capped at 8,000 words.
Results and Analysis
The S2SPMN models, particularly the dynamic version (DPMN), demonstrated superior performance across both automated and human metrics.
Quantitative Performance
DPMN100 achieved a test Perplexity (PPL) of 7.69, a notable improvement over the 9.05 PPL of the standard S2SA baseline. While the baseline scored higher on "Distinct" metrics (word variety), qualitative analysis revealed that this variety often consisted of irrelevant or ungrammatical tokens, highlighting the limitations of automated diversity scores in dialogue tasks.
Human Evaluation
Human annotators performed blind evaluations, scoring responses on a scale of 0 (irrelevant) to 2 (natural and relevant).
As shown in the evaluation results, DPMN100 significantly outperformed the baselines, with 31.25% of its responses receiving the highest rating, compared to only 6.83% for the standard S2SA model. This confirms that the prototype-guided approach effectively "nudges" the decoder away from generic replies toward more contextually appropriate language.
Key Insights
- Responses as Better Prototypes: The authors found that using responses as prototypes is more effective than using similar posts, as they provide a direct template for the required meaning transformation.
- Dynamic vs. Static: Dynamic retrieval is significantly more powerful; a DPMN using only 100 retrieved prototypes outperformed an SPMN using 1,000 random prototypes, suggesting that relevance is more critical than memory size.
Conclusion
S2SPMN provides a simple yet effective method for improving chatbot response quality by leveraging existing dialogue data as a memory source. The study highlights that the relevance of retrieved prototypes is more critical than the sheer size of the memory bank. Future work will explore more sophisticated semantic retrieval methods and techniques to further bridge the semantic gap in meaning transformation.
Summary updated