Introduction
In the field of Natural Language Processing (NLP), there is a recurring trend toward increasing architectural complexity. However, research from the University of Copenhagen suggests that a well-tuned, relatively simple model can often outperform sophisticated "state-of-the-art" systems. This paper demonstrates that a basic feed-forward network, when integrated into a multi-task learning (MTL) framework, provides a robust baseline for community question-answering (CQA) tasks.
Problem Statement: Question Relevancy Ranking
CQA platforms like StackOverflow and Quora frequently suffer from information clutter, including duplicate or irrelevant questions. The core challenge addressed is Question-Question Similarity: given a user's query, the system must accurately rank existing questions by their relevancy. Historically, top-performing systems relied on extensive hand-engineered pipelines but often struggled to surpass simple Information Retrieval (IR) baselines, such as standard search engine results. The authors set out to create a fast, language-independent, and easily reproducible baseline to set a higher bar for the field.
Methodology
The proposed approach moves away from complex Recurrent Neural Networks (RNNs) in favor of a Multi-Layer Perceptron (MLP) within an MTL architecture.
Feature Engineering
Instead of raw text, the model utilizes 14 distance measures calculated across three sentence representations: averaged word embeddings, binary unigrams, and trigrams. These measures include Cosine, Manhattan, Bhattacharya, and Euclidean distances, alongside the Jaccard Index for n-grams.
Architecture
The model employs a shared hidden layer that branches into task-specific layers. This design enables the network to learn general features of sentence similarity from auxiliary tasks while specializing in the primary ranking task. This was compared against a more complex bidirectional LSTM (Bi-LSTM) baseline to evaluate the utility of additional parameters.
Experimental Setup
The model was evaluated using the SemEval 2016 and 2017 (Subtask B) datasets. To enhance performance through MTL, several auxiliary tasks were incorporated to provide additional signals for sentence relationship modeling:
- SemEval Subtask A: Ranking comments relative to questions.
- Fake News Challenge (FNC): Determining the stance of a body of text relative to a headline.
- Multi-Genre Natural Language Inference (MNLI): Predicting entailment or contradiction between sentence pairs.
Results and Discussion
The MTL-MLP model significantly outperformed both the complex Bi-LSTM baseline and previous competition winners across multiple metrics.
As shown in the results, the MTL-MLP model achieved a Mean Average Precision (MAP) of 77.61 on the SemEval-2016 dataset, surpassing the previous "SemEval-Best" of 76.70. Furthermore, it consistently outperformed the IR baseline, a feat many complex neural models have failed to achieve.
Key insights from the experiments include:
- The Efficacy of MTL: Training on auxiliary tasks like fake news detection (FNC) and inference (MNLI) provides a "weak signal" that improves generalization, especially when the primary dataset is small.
- Simplicity vs. Complexity: The MLP consistently beat the Bi-LSTM. The authors suggest that high-parameter models like LSTMs are more prone to overfitting and are harder to tune for tasks with limited data.
- Feature Robustness: Ablation studies indicated that basic lexical overlap features, such as the Jaccard index and Cosine similarity over unigrams and trigrams, carry a surprisingly high amount of signal for retrieval tasks.
Conclusion
By demonstrating that a simple MLP with 14 distance features can outperform complex recurrent networks, this work provides a vital sanity check for the NLP community. It establishes a strong, language-independent, and reproducible baseline that emphasizes the importance of foundational model design and the strategic use of multi-task learning before moving toward more complex architectures.
Summary updated