A Simple Approach to Jointly Rank Passages and Select Relevant Sentences in the OBQA Context

Man Luo, Shuguang Chen, Chitta Baral

Introduction

Open Book Question Answering (OBQA) requires systems to not only provide an answer but also to retrieve the specific evidence (supporting facts) from a large corpus. Datasets like HotpotQA emphasize this by requiring multi-hop reasoning across multiple documents. Traditionally, this has been addressed using complex, multi-stage pipelines. For instance, one large-scale model might rank passages for relevance, while a second, independent model extracts specific sentences from those passages.

While effective, these decoupled pipelines are computationally expensive and fail to exploit the inherent relationship between passage-level and sentence-level relevance. This paper introduces the "Two-in-One" framework, a single, lightweight transformer model designed to perform both tasks simultaneously, promoting computational efficiency and task synergy.

The Two-in-One Framework

The core of the proposed approach is a single RoBERTa encoder that processes the question and candidate passages in a unified pass.

The Two-in-One architecture jointly processes queries and passages to output both passage ranks and sentence classifications.

Shared Representation and Multi-Task Heads

As shown in the architecture diagram, the input is formatted by concatenating the question (Q) with the sentences (S1, S2, ..., Sk) of a passage. Special tokens are strategically placed: the <s> token at the start captures the global context of the passage, while </s> tokens following each sentence capture local sentence-level context. Two Multi-Layer Perceptron (MLP) heads are then applied to these representations: one for passage ranking and another for sentence classification.

Enforcing Logical Constraints

To ensure the model's predictions are logically consistent, the authors introduce two novel constraints:

  1. Consistency Constraint: This loss function minimizes the difference between the passage relevance score and the maximum relevance score of the sentences within that passage. This enforces the intuition that a relevant passage must contain at least one highly relevant sentence.
  2. Similarity Constraint: Using a triplet loss, the model is trained to ensure the vector representation of a relevant passage (the <s> token) is closer to its supporting sentences than to irrelevant ones.

Experimental Setup

The model was evaluated on the HotpotQA Distractor setting. In this benchmark, the system is provided with a question and 10 candidate passages (containing both gold standard passages and "hard" distractors). The goal is to identify the exact supporting sentences required to answer the question.

Results and Discussion

The Two-in-One model demonstrates that joint learning significantly outperforms single-task baselines while remaining competitive with much larger state-of-the-art models.

Performance comparison showing the Two-in-One model achieving competitive results with significantly fewer parameters than SOTA pipelines.

Performance and Efficiency

As detailed in the results table, the Two-in-One model achieved a 59.17% Exact Match (EM) for supporting facts, a massive improvement over the ~28% EM achieved by single-task baseline models. Furthermore, with an 85.82% F1 score, it performs comparably to the QUARK pipeline (86.97% F1). Crucially, the Two-in-One model achieves this using only ~330M parameters, roughly one-third of the ~1020M parameters required by QUARK. This highlights the model's alignment with "Green AI" principles by reducing the computational footprint without sacrificing accuracy.

Ablation Insights

Ablation studies confirmed the efficacy of the constraints. The consistency constraint successfully reduced the prediction gap between passage and sentence scores from 0.11 to 0.03, proving that the model learned to logically align its multi-level predictions.

Conclusion

The Two-in-One framework proves that joint passage ranking and sentence selection is not only more efficient but also more effective than decoupled pipelines. By leveraging shared representations and logical constraints, the model achieves state-of-the-art performance with a fraction of the parameters.

Future work aims to extend this into a "Three-in-One" model by adding a third head for final answer span extraction and testing the system's performance in full open-domain settings where millions of documents must be searched.

Summary updated

BibTeX
@inproceedings{luo2022simple,
  author = {Man Luo and Shuguang Chen and Chitta Baral},
  title = {{A Simple Approach to Jointly Rank Passages and Select Relevant Sentences in the OBQA Context}},
  booktitle = {Student Research Workshop @ NAACL},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-srw.23},
}
Figure