Hierarchical Ranking for Answer Selection

Hang Gao, Mengting Hu, Renhong Cheng, Tiegang GaoarXiv 2021 · arXiv

Introduction

Answer selection is a core component of question-answering (QA) systems, requiring models to identify the most relevant answer from a set of candidates. While various ranking strategies exist, they are often used in isolation. This paper proposes a hierarchical ranking strategy that integrates point-level, pair-level, and list-level objectives to capture diverse granularities of relevance. By progressively fusing these objectives, the model achieves state-of-the-art performance among non-pretrained architectures on standard benchmarks.

Problem Definition: Ranking Granularities

Traditionally, neural ranking models for answer selection follow one of three paradigms:

  1. Point-level: Treats the task as binary classification, determining if a specific answer is relevant to the question.
  2. Pair-level: Uses contrastive learning to determine if one answer is better than another.
  3. List-level: Optimizes the global ranking of all candidate answers simultaneously.
The three levels of ranking granularity: point-level, pair-level, and list-level.

While these levels are complementary, standard Multi-Task Learning (MTL) often fails to exploit the inherent relationships between them. The authors address this by investigating how to effectively fuse these objectives to mutually reinforce the model's learning process.

Proposed Methods

Using the Compare-Aggregate Network as a backbone, the authors explore three integration schemes:

1. Multi-Task Learning (MTL)

This baseline shares encoding and interaction layers but uses separate comparison/aggregation branches for each loss (point, pair, and list), optimized simultaneously.

2. Ranking Integration (RI)

RI designates one level as the primary objective and uses the others as auxiliary features. For example, in RI(list), features from point and pair levels are concatenated to the list-level representation before final prediction.

3. Progressive Ranking Integration (PRI)

The most innovative approach, PRI, follows a "divide-and-conquer" philosophy. In the PRI(list) variant, the model builds representations hierarchically: it first extracts point-level features, combines them to form pair-level features, and finally integrates those into a comprehensive list-level representation.

Architecture of the Progressive Ranking Integration (PRI) scheme for the list-level objective.

Experimental Results

The models were evaluated on two public datasets: TREC-QA (Clean version) and WikiQA.

Key Performance Metrics

  • WikiQA: The PRI(list) model achieved a Mean Average Precision (MAP) of 0.742 and a Mean Reciprocal Rank (MRR) of 0.754.
  • TREC-QA: The model reached a MAP of 0.841 and an MRR of 0.898.

These results represent the state-of-the-art for non-pretrained models. Ablation studies confirmed that removing either the point-level or pair-level components from the PRI scheme led to significant performance degradation, validating the necessity of the full hierarchy.

Key Insights

  • Complementary Features: Point-level ranking captures absolute relevance, pair-level captures contrastive differences, and list-level captures global structure. Integrating them provides a more holistic view of the candidate pool.
  • Efficiency: Jointly training these hierarchical objectives acts as a regularizer, leading to faster convergence compared to models trained on a single list-level objective.
  • The Pre-training Gap: While PRI(list) leads the non-pretrained field, large-scale pretrained models like RoBERTa still maintain a significant lead (e.g., 0.936 MAP on TREC-QA).

Conclusion

The hierarchical ranking strategy, particularly the Progressive Ranking Integration (PRI) scheme, demonstrates that explicitly modeling the relationships between different data granularities significantly improves answer selection. Future research may involve applying these hierarchical principles to modern Transformer-based architectures to further bridge the gap with large-scale pre-training.

Summary updated

Abstract

Answer selection is a task to choose the positive answers from a pool of candidate answers for a given question. In this paper, we propose a novel strategy for answer selection, called hierarchical ranking. We introduce three levels of ranking: point-level ranking, pair-level ranking, and list-level ranking. They formulate their optimization objectives by employing supervisory information from different perspectives to achieve the same goal of ranking candidate answers. Therefore, the three levels of ranking are related and they can promote each other. We take the well-performed compare-aggregate model as the backbone and explore three schemes to implement the idea of applying the hierarchical rankings jointly: the scheme under the Multi-Task Learning (MTL) strategy, the Ranking Integration (RI) scheme, and the Progressive Ranking Integration (PRI) scheme. Experimental results on two public datasets, WikiQA and TREC-QA, demonstrate that the proposed hierarchical ranking is effective. Our method achieves state-of-the-art (non-BERT) performance on both TREC-QA and WikiQA.
BibTeX
@misc{gao2021hierarchical,
  author = {Hang Gao and Mengting Hu and Renhong Cheng and Tiegang Gao},
  title = {{Hierarchical Ranking for Answer Selection}},
  year = {2021},
  eprint = {2102.00677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.00677},
}
Figure