VectorSLU: A Continuous Word Vector Approach to Answer Selection in Community Question Answering Systems

Yonatan Belinkov, Mitra Mohtarami, Scott Cyphers, James R. Glass

Abstract

Community Question Answering (CQA) platforms are valuable knowledge repositories but often suffer from inconsistent answer quality. VectorSLU is a system designed to automatically evaluate answer quality using continuous word and phrase vectors. By combining traditional text-similarity metrics with deep semantic features derived from Word2Vec and Doc2Vec, the system achieved state-of-the-art results in the SemEval-2015 competition, notably securing first place in the English YES/NO inference task.

Introduction

Community Question Answering (CQA) services, such as Quora or Stack Overflow, provide a wealth of information but face significant challenges due to the high variance in response quality. A single query can elicit numerous responses, ranging from definitive solutions to irrelevant chatter. This paper addresses two primary tasks:

  1. Answer Selection: Categorizing answers as good (complete), potential (useful but incomplete), or bad (irrelevant).
  2. YES/NO Inference: Determining if the collective "good" answers to a polar question imply a yes, no, or unsure response, even in the absence of explicit keywords.

The core challenge lies in the nuance of human language; answers may be semantically relevant without sharing exact keywords, rendering traditional matching techniques insufficient.

Methodology

The VectorSLU system utilizes a multi-class linear Support Vector Machine (SVM) classifier. The researchers engineered a diverse feature set to capture both literal and semantic relationships between questions and answers.

Overview of the feature categories used in VectorSLU, including text-based, vector-based, metadata, and rank-based features.

The feature set includes:

  • Vector-Based Features: Utilizing Word2Vec and Doc2Vec to represent words and entire documents as dense vectors. This allows the system to recognize semantic relationships (e.g., "driving license" and "permit").
  • Text-Based Similarities: Classic NLP metrics such as Longest Common Substring, Jaccard coefficient, and Greedy String Tiling to measure literal overlap.
  • Metadata & Rank-Based Features: Tracking user identity (e.g., if the asker is also the answerer) and using SVM Rank to provide relative quality scores.
  • Sentiment/Polarity Expansion: For YES/NO tasks, the system identifies the nearest neighbors in vector space for "yes," "no," and "probably" to capture a broader range of affirmative and negative cues.

Experimental Setup

The system was evaluated using the SemEval-2015 Task 3 datasets:

  • English CQA: 2,600 training questions and over 16,000 answers.
  • Arabic CQA: 1,300 training questions.
  • Pre-trained Vectors: English models used the GoogleNews dataset (3 million words), while Arabic models were trained on the Arabic Gigaword corpus.

Key Insights

A significant finding was that feeding normalized vectors themselves into the SVM outperformed the standard use of a single cosine similarity score. This allows the classifier to learn which specific dimensions of the vector space are most critical for determining quality. Additionally, the researchers found that computing average vectors for individual sentences or "chunks" provided better granularity than a simple "bag of words" approach. The system's reliance on unsupervised word vectors also made it remarkably portable across languages, performing well on both English and Arabic with minimal tuning.

Results

VectorSLU proved highly competitive in the SemEval-2015 benchmarks. Most notably, it achieved the top rank in the English YES/NO inference task.

Performance comparison for the English YES/NO inference task, showing VectorSLU-Primary achieving the highest Macro-F1 and Accuracy scores.

Summary of rankings:

  • English YES/NO Inference: 1st out of 8 submissions (Macro-F1: 63.70).
  • Arabic Answer Selection: 2nd out of 4 submissions (Macro-F1: 73.18).
  • English Answer Selection: 6th out of 12 submissions (Macro-F1: 49.54).

Conclusion

The success of VectorSLU demonstrates that while text-overlap features are useful, continuous vectors provide the semantic "glue" necessary to understand complex relationships in community forums. While the current work focuses on linear SVMs, the results lay the groundwork for future exploration into deep neural architectures like LSTMs or Transformers to better capture the sequential nature of dialogue.

Summary updated

BibTeX
@inproceedings{belinkov2015vectorslu,
  author = {Yonatan Belinkov and Mitra Mohtarami and Scott Cyphers and James R. Glass},
  title = {{VectorSLU: A Continuous Word Vector Approach to Answer Selection in Community Question Answering Systems}},
  booktitle = {SemEval @ NAACL},
  year = {2015},
  doi = {10.18653/v1/s15-2048},
}
Figure