A strong baseline for question relevancy ranking

Ana Valeria González-Garduño, Isabelle Augenstein, Anders Søgaard

Introduction

In the field of Natural Language Processing (NLP), there is a recurring trend toward increasing architectural complexity. However, research from the University of Copenhagen suggests that a well-tuned, relatively simple model can often outperform sophisticated "state-of-the-art" systems. This paper demonstrates that a basic feed-forward network, when integrated into a multi-task learning (MTL) framework, provides a robust baseline for community question-answering (CQA) tasks.

Problem Statement: Question Relevancy Ranking

CQA platforms like StackOverflow and Quora frequently suffer from information clutter, including duplicate or irrelevant questions. The core challenge addressed is Question-Question Similarity: given a user's query, the system must accurately rank existing questions by their relevancy. Historically, top-performing systems relied on extensive hand-engineered pipelines but often struggled to surpass simple Information Retrieval (IR) baselines, such as standard search engine results. The authors set out to create a fast, language-independent, and easily reproducible baseline to set a higher bar for the field.

Methodology

The proposed approach moves away from complex Recurrent Neural Networks (RNNs) in favor of a Multi-Layer Perceptron (MLP) within an MTL architecture.

Feature Engineering

Instead of raw text, the model utilizes 14 distance measures calculated across three sentence representations: averaged word embeddings, binary unigrams, and trigrams. These measures include Cosine, Manhattan, Bhattacharya, and Euclidean distances, alongside the Jaccard Index for n-grams.

Architecture

The model employs a shared hidden layer that branches into task-specific layers. This design enables the network to learn general features of sentence similarity from auxiliary tasks while specializing in the primary ranking task. This was compared against a more complex bidirectional LSTM (Bi-LSTM) baseline to evaluate the utility of additional parameters.

The Multi-Task Learning (MTL) architecture featuring a shared hidden layer for joint feature learning across main and auxiliary tasks.

Experimental Setup

The model was evaluated using the SemEval 2016 and 2017 (Subtask B) datasets. To enhance performance through MTL, several auxiliary tasks were incorporated to provide additional signals for sentence relationship modeling:

  • SemEval Subtask A: Ranking comments relative to questions.
  • Fake News Challenge (FNC): Determining the stance of a body of text relative to a headline.
  • Multi-Genre Natural Language Inference (MNLI): Predicting entailment or contradiction between sentence pairs.

Results and Discussion

The MTL-MLP model significantly outperformed both the complex Bi-LSTM baseline and previous competition winners across multiple metrics.

Comparative performance of STL and MTL models across SemEval-16 and SemEval-17 benchmarks, showing the MTL-MLP model surpassing previous bests.

As shown in the results, the MTL-MLP model achieved a Mean Average Precision (MAP) of 77.61 on the SemEval-2016 dataset, surpassing the previous "SemEval-Best" of 76.70. Furthermore, it consistently outperformed the IR baseline, a feat many complex neural models have failed to achieve.

Key insights from the experiments include:

  1. The Efficacy of MTL: Training on auxiliary tasks like fake news detection (FNC) and inference (MNLI) provides a "weak signal" that improves generalization, especially when the primary dataset is small.
  2. Simplicity vs. Complexity: The MLP consistently beat the Bi-LSTM. The authors suggest that high-parameter models like LSTMs are more prone to overfitting and are harder to tune for tasks with limited data.
  3. Feature Robustness: Ablation studies indicated that basic lexical overlap features, such as the Jaccard index and Cosine similarity over unigrams and trigrams, carry a surprisingly high amount of signal for retrieval tasks.

Conclusion

By demonstrating that a simple MLP with 14 distance features can outperform complex recurrent networks, this work provides a vital sanity check for the NLP community. It establishes a strong, language-independent, and reproducible baseline that emphasizes the importance of foundational model design and the strategic use of multi-task learning before moving toward more complex architectures.

Summary updated

Abstract

The best systems at the SemEval-16 and SemEval-17 community question answering shared tasks -- a task that amounts to question relevancy ranking -- involve complex pipelines and manual feature engineering. Despite this, many of these still fail at beating the IR baseline, i.e., the rankings provided by Google's search engine. We present a strong baseline for question relevancy ranking by training a simple multi-task feed forward network on a bag of 14 distance measures for the input question pair. This baseline model, which is fast to train and uses only language-independent features, outperforms the best shared task systems on the task of retrieving relevant previously asked questions.
BibTeX
@inproceedings{gonzalezgarduno2018strong,
  author = {Ana Valeria González-Garduño and Isabelle Augenstein and Anders Søgaard},
  title = {{A strong baseline for question relevancy ranking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure