Abstract
Community Question Answering (CQA) systems often struggle with long-term semantic matching, where the relationship between a query and its potential answer is obscured by distance or lack of context. This paper presents a novel architecture, Memory-aware attentive control with knowledge-based dual refinement (MMN), designed to address these limitations. By integrating internal knowledge mining via topic-specific memory pools and a unique two-dimensional dual refinement mechanism, MMN effectively bridges semantic gaps. The model achieves state-of-the-art results on benchmark datasets, including WikiQA and SQuAD, demonstrating superior performance in both answer selection and machine reading comprehension.
Introduction
In the realm of Community Question Answering (CQA), the ability to accurately match questions with relevant answers is paramount. While modern Transformer-based models like BERT and RoBERTa have revolutionized natural language processing, they often encounter difficulties with long-term semantic dependencies. This paper, published in IEEE Transactions on Systems, Man, and Cybernetics: Systems, introduces the MMN model. MMN leverages "memory pools" of related questions to prime its understanding and employs a dual-writing mechanism to iteratively refine semantic connections, navigating the nuances of human language more effectively than traditional approaches.
The Challenge of Long-Term Semantic Matching
CQA systems typically perform two main tasks: Answer Selection (AS), choosing the best answer from a candidate pool, and Machine Reading Comprehension (MRC), extracting a specific text span from a document. The primary hurdle in both is long-term semantic matching.
Existing models often struggle when an answer is buried deep within a long document or when a question is too brief to provide sufficient context. While previous memory-based networks attempted to mitigate this, they were frequently hampered by fixed refinement mechanisms or the loss of vital information over multiple iterations. MMN addresses this by creating a robust internal knowledge system that moves beyond analyzing question-answer pairs in isolation.
Proposed Methodology: The MMN Model
The MMN architecture introduces three core innovations to improve semantic matching:
1. Knowledge-Based Memory Initialization
Rather than starting from a random state, MMN utilizes "Memory Pools." The model mines the existing corpus for the top-k most similar question-answer pairs related to the current topic. This "prior knowledge" helps the system interpret ambiguous or short queries by providing a broader context.
2. Dual Refinement Mechanism
At the heart of MMN is the dual refinement mechanism. Unlike standard memory networks that update state in a single pass, MMN uses two distinct "writing heads":
- Row-based attention: Refines memory based on the answer or context.
- Column-based attention: Refines memory based on the question.
This iterative, two-dimensional approach captures "multi-view" salient word pairs, allowing the question and answer to constantly re-inform each other.
3. Top-k Max Pooling
To maintain computational efficiency, MMN employs a pooling layer that aggregates only the most significant word-level similarities. By filtering out irrelevant "noise" and using ReLU activations, the model maintains a sparse memory matrix, enabling it to handle longer contexts without exponential complexity growth.
Experimental Evaluation
The researchers evaluated MMN across four major public datasets: TREC and WikiQA for Answer Selection, and SQuAD and TriviaQA for Machine Reading Comprehension. The model was further enhanced through transfer learning, pre-training on the massive StackExchangeQA dataset before fine-tuning on these specific benchmarks.
Key Results and Insights
MMN consistently outperformed or matched state-of-the-art baselines across all tested datasets:
- WikiQA: The RoBERTa-Large version of MMN achieved a Mean Average Precision (MAP) of 0.9323, surpassing the highly regarded TANDA model.
- SQuAD: MMN reached an F1 score of 87.3%, reaching parity with complex readers like M-Reader.
- TriviaQA: The model achieved a 73.57% F1 score, a significant improvement over the 71.39% achieved by the second-best model, SLQA.
Ablation studies confirmed that "Dual Refinement" significantly outperformed single-dimensional models, suggesting that semantic matching is inherently a bilateral process. Furthermore, the "warm" initialization provided by memory pools proved critical to the model's success.
Conclusion and Future Work
The MMN model represents a significant step forward in CQA by effectively utilizing internal knowledge and iterative refinement. While highly effective, the authors acknowledge challenges such as the computational resources required for training and the need to truncate extremely long documents. Future research may explore reinforcement learning to optimize memory retention and forgetting processes over even longer sequences, potentially eliminating the need for context truncation.
Summary updated