Introduction
Answer selection is a core component of question-answering (QA) systems, requiring models to identify the most relevant answer from a set of candidates. While various ranking strategies exist, they are often used in isolation. This paper proposes a hierarchical ranking strategy that integrates point-level, pair-level, and list-level objectives to capture diverse granularities of relevance. By progressively fusing these objectives, the model achieves state-of-the-art performance among non-pretrained architectures on standard benchmarks.
Problem Definition: Ranking Granularities
Traditionally, neural ranking models for answer selection follow one of three paradigms:
- Point-level: Treats the task as binary classification, determining if a specific answer is relevant to the question.
- Pair-level: Uses contrastive learning to determine if one answer is better than another.
- List-level: Optimizes the global ranking of all candidate answers simultaneously.
While these levels are complementary, standard Multi-Task Learning (MTL) often fails to exploit the inherent relationships between them. The authors address this by investigating how to effectively fuse these objectives to mutually reinforce the model's learning process.
Proposed Methods
Using the Compare-Aggregate Network as a backbone, the authors explore three integration schemes:
1. Multi-Task Learning (MTL)
This baseline shares encoding and interaction layers but uses separate comparison/aggregation branches for each loss (point, pair, and list), optimized simultaneously.
2. Ranking Integration (RI)
RI designates one level as the primary objective and uses the others as auxiliary features. For example, in RI(list), features from point and pair levels are concatenated to the list-level representation before final prediction.
3. Progressive Ranking Integration (PRI)
The most innovative approach, PRI, follows a "divide-and-conquer" philosophy. In the PRI(list) variant, the model builds representations hierarchically: it first extracts point-level features, combines them to form pair-level features, and finally integrates those into a comprehensive list-level representation.
Experimental Results
The models were evaluated on two public datasets: TREC-QA (Clean version) and WikiQA.
Key Performance Metrics
- WikiQA: The PRI(list) model achieved a Mean Average Precision (MAP) of 0.742 and a Mean Reciprocal Rank (MRR) of 0.754.
- TREC-QA: The model reached a MAP of 0.841 and an MRR of 0.898.
These results represent the state-of-the-art for non-pretrained models. Ablation studies confirmed that removing either the point-level or pair-level components from the PRI scheme led to significant performance degradation, validating the necessity of the full hierarchy.
Key Insights
- Complementary Features: Point-level ranking captures absolute relevance, pair-level captures contrastive differences, and list-level captures global structure. Integrating them provides a more holistic view of the candidate pool.
- Efficiency: Jointly training these hierarchical objectives acts as a regularizer, leading to faster convergence compared to models trained on a single list-level objective.
- The Pre-training Gap: While PRI(list) leads the non-pretrained field, large-scale pretrained models like RoBERTa still maintain a significant lead (e.g., 0.936 MAP on TREC-QA).
Conclusion
The hierarchical ranking strategy, particularly the Progressive Ranking Integration (PRI) scheme, demonstrates that explicitly modeling the relationships between different data granularities significantly improves answer selection. Future research may involve applying these hierarchical principles to modern Transformer-based architectures to further bridge the gap with large-scale pre-training.
Summary updated