TL;DR
This paper introduces a surprisingly effective and efficient baseline for question relevancy ranking, a crucial task in community question answering. By employing a simple multi-task learning approach with a feed-forward network and a set of carefully chosen distance features, they achieve state-of-the-art results, outperforming complex, hand-engineered systems, especially in low-data scenarios.
What Problem Does This Paper Address?
Community Question Answering (CQA) platforms are rich sources of information, but they often suffer from a deluge of irrelevant answers. The task of question relevancy ranking aims to address this by identifying which previously asked questions are most relevant to a new query. This is a challenging retrieval problem, and despite advancements, many systems struggle to outperform a simple Google search baseline. The SemEval CQA shared tasks have highlighted this difficulty, particularly for the question-question similarity subtask, which often has limited training data.
Methods
The core of the paper's contribution is a simple yet powerful Multi-Task Learning (MTL) architecture. They propose a feed-forward neural network (MLP) that takes a pair of questions as input. Instead of complex linguistic features, the input consists of 14 distance measures calculated between different representations of the question pair. These representations include:
- Sentence Representations:
- Averaged word embeddings
- Binary unigram vectors
- Trigram vectors
- Similarity Metrics:
- Cosine distance
- Manhattan distance
- Bhattacharya distance
- Euclidean distance
- Jaccard index (applicable only to unigram/trigram vectors)
The MLP has a shared hidden layer that is trained on both the main question relevancy ranking task and an auxiliary task. This auxiliary task helps the model learn more robust representations and generalize better, especially when labeled data for the main task is scarce. They experiment with several auxiliary tasks, including question-comment relevancy prediction, fake news detection, and natural language inference.
Datasets
The primary dataset used for the main task is from the SemEval-2016 and SemEval-2017 Task 3 (Community Question Answering), specifically subtask B, which involves ranking potentially related questions. For auxiliary tasks, they utilize:
- SemEval-2016 and SemEval-2017 Task 3 subtask A: Question-comment relevancy ranking.
- Fake News Challenge (FNC): Classifying the stance of a body of text relative to a headline.
- Multi-Genre Natural Language Inference (MultiNLI): Predicting entailment, contradiction, or neutrality between a premise and a hypothesis.
Key Insights and Ideas
- Simplicity Trumps Complexity: The paper powerfully demonstrates that a straightforward feed-forward network with well-chosen distance features can outperform complex recurrent neural networks and heavily engineered systems. This is a significant finding, suggesting that the problem might be more about effective feature representation than intricate model architectures.
- The Power of Multi-Task Learning: MTL proves to be highly effective, especially in low-data regimes. By training on auxiliary tasks, the model learns more generalizable representations, leading to improved performance on the main question relevancy task. This is particularly evident in the learning curves, where MTL models show more stable learning across different training set sizes.
- Distance Features are Key: The selection of 14 distance measures, derived from various sentence representations, is crucial. These features capture different aspects of semantic and syntactic similarity without requiring deep linguistic analysis.
- Language-Independent Features are Sufficient: The reliance on language-independent features (like bag-of-words or embeddings) makes the model adaptable and efficient, avoiding the need for language-specific processing pipelines.
Key Results
- The proposed MTL-MLP model outperforms the best SemEval systems for both SemEval-2016 and SemEval-2017 on the question-question similarity task, achieving higher Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR).
- The MTL approach significantly improves performance over single-task learning (STL) baselines, with p-values less than 0.01.
- The MTL-MLP model also outperforms the official IR baseline (Google search) used in the SemEval tasks.
- Even a bidirectional LSTM baseline augmented with the same distance features does not outperform the simpler MTL-MLP, highlighting the effectiveness of the feature set and MTL approach.
- Learning curves show that MTL models are more stable and achieve better performance with fewer training samples compared to STL models.
Limitations and Future Work
- Limited Feature Set: While effective, the paper focuses on a specific set of 14 distance features. Future work could explore richer feature sets, including those derived from knowledge graphs or frame semantics.
- Ablation Study Insights: The ablation study (Table 3) reveals that removing Euclidean distance over trigrams and cosine similarity over unigrams most negatively impacts performance, suggesting these are particularly important features. The Jaccard index over unigrams also shows a strong signal.
- Model Complexity vs. Performance: The paper acknowledges that while their model is simpler to implement and train than some state-of-the-art systems, further exploration into more complex architectures that leverage their feature set could be beneficial.
- Generalizability of Auxiliary Tasks: The paper demonstrates the utility of various auxiliary tasks. Future work could investigate which types of auxiliary tasks are most beneficial for question relevancy ranking and explore even more diverse tasks.
Summary updated