Introduction
In community Question Answering (cQA) forums like StackExchange, answers are often long, multi-paragraph explanations. Selecting the correct answer from a pool of candidates is a "needle in a haystack" problem. Traditional neural models struggle with three main issues: the "fluff" problem where relevant signals are drowned out by noise in long answers, data scarcity in niche communities, and a lack of syntactic awareness.
This paper introduces COALA (COverage-based Approach for Long Answer selection), a neural architecture that shifts the focus from measuring how much of an answer is relevant to measuring how much of the question is covered by the answer. This approach allows the model to outperform state-of-the-art systems on long-form answers while remaining effective even in low-resource environments.
Methodology
COALA utilizes a Compare-Aggregate framework but re-engineers the aggregation step to prioritize question coverage rather than answer-wide relevance.
The process involves several key stages:
- Aspect Identification: Instead of data-hungry LSTMs, COALA uses a simple CNN to extract local n-gram "aspects" from the question and answer.
- Interaction Matrix: A dot-product matrix ($H$) represents the similarity between every question aspect and every answer aspect.
- Coverage-Based Matching: For every aspect of the question, the model finds the best matching aspect in the answer via max-pooling. This determines if a specific part of the question is addressed anywhere in the answer.
- Aggregation: The model averages these coverage scores. By aggregating over question aspects (which are typically few) rather than answer aspects (which can be many), the model remains invariant to answer length.
The authors also proposed variants like COALA p-means for complex statistical summaries and COALA syntax-aware, which incorporates dependency parse tree embeddings to provide linguistic context.
Experimental Setup and Datasets
COALA was evaluated on standard benchmarks like InsuranceQA (v2) and WikiPassageQA, as well as five newly curated StackExchange datasets (Travel, Cooking, Academia, Apple, and Aviation). These datasets were chosen to test real-world cQA scenarios where answers are long and training data is often scarce.
Key Results
COALA demonstrates significant improvements over state-of-the-art models, particularly in challenging environments.
Small Data Robustness
COALA is remarkably effective in "small data" scenarios. When trained on only 25 questions, it achieved nearly 35% accuracy, whereas complex models like CA-Wang struggled to reach 5%, performing significantly worse than simple unsupervised baselines like TF*IDF.
Performance on Long Text
The model's question-centric approach ensures that "fluff" in long answers does not degrade performance. COALA outperformed the previous state-of-the-art by 21 percentage points on answers longer than 250 words. Across six cQA datasets, COALA improved accuracy by an average of 4.5pp over existing models.
The Value of Syntax
Even in informal cQA text, adding syntactic dependency information consistently improved results. This suggests that linguistic structure helps the model disambiguate similar-looking candidate answers that might otherwise fool a bag-of-words approach.
Discussion and Limitations
While COALA excels at long answers and low-resource settings, it is slightly less effective for very short answers (under 100 words). In these cases, the lack of noise allows models that process global context to perform better.
Future work may involve a two-step ranking process: using COALA to efficiently filter a pool of high-coverage candidates, followed by a more computationally intensive, context-aware model to make the final selection. This hybrid approach could leverage COALA's efficiency and robustness while capturing finer semantic nuances for shorter texts.
Summary updated