Neural Passage Quality Estimation for Static Pruning

Xuejun Chang, Debabrata Mishra, Craig Macdonald, Sean MacAvaneyarXiv 2024 · DOI · arXiv

TL;DR

Researchers from the University of Glasgow have developed a way to prune search engine indexes by identifying and removing "low-quality" passages that are unlikely to ever answer a user's query. By using a lightweight supervised neural model called QT5, they can discard over 25% of a document corpus while maintaining the same search effectiveness, significantly reducing the computational costs, storage requirements, and carbon footprint of modern neural search engines.

What Problem Does This Paper Address?

Modern search engines, particularly those using "Neural IR" (Information Retrieval), are incredibly powerful but computationally expensive. Models like dense retrievers (e.g., TAS-B) or learned sparse models (e.g., SPLADE) require significant resources to encode, store, and search through millions—or billions—of text passages.

The standard assumption in IR is that the corpus is fixed: we index everything and then figure out what's relevant to a specific query. However, many passages in a document are essentially "noise"—discourse, repetitive headers, or filler text that will likely never satisfy any user's information need. The authors ask: Can we identify these low-quality passages before indexing? If we can prune them early, we save on every downstream cost: indexing time, storage space, and retrieval latency.

Methods

The authors explore several ways to estimate "passage quality" (query-agnostic relevance):

  1. Statistical Baselines: Simple lexical measures like the Information-To-Noise (ITN) ratio (unique terms vs. total terms) and Collection-Document Distance (CDD).
  2. Unsupervised Neural Estimators: Using the perplexity of a pre-trained language model (like GPT-2). The hypothesis is that "weird" or low-likelihood text sequences are likely low quality.
  3. Latent Neural Estimators: Extracting signals already present in trained retrieval models, such as the magnitude of a dense vector or specific quality weights from models like EPIC.
  4. Supervised Neural Estimators (QT5): The authors' primary contribution. They fine-tune a T5 model to predict if a passage is relevant to any query. Crucially, they train it on standard relevance triples (Query, Positive Passage, Negative Passage) but strip away the query, forcing the model to learn what a "good" passage looks like in isolation.

Datasets

The study uses several well-established benchmarks to ensure the findings generalize across scales and domains:

  • MSMARCO v1 (Passage): The primary dataset for training and initial evaluation.
  • MSMARCO v2: A much larger corpus (138 million passages) to test scalability.
  • CORD-19: A dataset of COVID-19 research papers to test domain transferability.
  • Evaluation Sets: TREC Deep Learning (DL) tracks from 2019 through 2022 and TREC COVID.

Key Insights and Ideas

  • Quality is Query-Agnostic: While IR usually focuses on $P(relevance | query, passage)$, this work successfully models $P(relevance | passage)$. Some text is just inherently more "useful" for search than others.
  • Supervision is Key: Simple statistical or unsupervised methods (like perplexity) struggle to beat a random pruning baseline. Direct supervision on relevance labels provides a much stronger signal.
  • Size Doesn't Always Matter: A "Tiny" 4-layer T5 model (QT5-Tiny) is nearly as effective at pruning as the much larger "Base" version. In fact, smaller models may be less prone to over-fitting the specific quirks of the training data.
  • Pruning as a Denoising Step: In some cases, like the CORD-19 dataset, pruning actually improved search precision. This suggests that low-quality passages aren't just a waste of space; they can be active distractors for retrieval models.
Figure 2: ROC curves for each passage quality estimator, based on a union of all relevant documents in the full MSMARCO dev set, DL 2019, and DL 2020 and excluding all relevant passages from the train set. The figure details the range [0.8,1.0], thereby focussing on the passages most likely to be pruned. The AUC for each estimator is in the legend.

Key Results

  • Effective Pruning: The QT5 model can consistently prune 25-30% of the MSMARCO corpus without any statistically significant drop in search effectiveness (measured by nDCG@10 and RR@10).
  • Efficiency Gains: Pruning 25% of a PISA BM25 index reduced the index size by 24.3% and retrieval time by 9.8%.
  • Indexing Speedup: For neural models, the "break-even" point for the QT5-Tiny model is around 14% pruning. If you prune more than that, the time saved by not having to encode those passages into dense/sparse vectors outweighs the time spent running the quality estimator.
  • Robustness: The findings held true across different retrieval paradigms, including lexical (BM25), dense (TAS-B), learned sparse (SPLADE), and cross-encoder re-rankers (MonoELECTRA).
Figure 3: Precision-oriented retrieval effectiveness on four pipelines by the percentage of a corpus pruned using each quality estimator. Effectiveness measurements that are statistically equivalent to the unpruned passage corpus are marked with . Note that the vertical axis of each plot are scaled to emphasise the effect on each individual model.

Limitations and Future Work

The authors identify several areas for further exploration:

  • Segmentation Integration: The current approach assumes passages are already segmented. A more advanced method could integrate quality estimation into the segmentation process itself, deciding where to split (or skip) text dynamically.
  • Advanced Training: The supervised model is currently trained "naively" on relevance labels. Techniques like knowledge distillation from a powerful teacher ranker could potentially improve the quality signal.
  • Compositionality: While this work focuses on passage pruning, it could be combined with other techniques like term pruning or vector pruning for even greater efficiency gains.
  • Edge Cases: Qualitative analysis showed that the model sometimes prunes repetitive text that might actually be useful (like song lyrics or directory listings). Better handling of these edge cases is needed.

Summary updated

Abstract

Neural networks -- especially those that use large, pre-trained language models -- have improved search engines in various ways. Most prominently, they can estimate the relevance of a passage or document to a user's query. In this work, we depart from this direction by exploring whether neural networks can effectively predict which of a document's passages are unlikely to be relevant to any query submitted to the search engine. We refer to this query-agnostic estimation of passage relevance as a passage's quality. We find that our novel methods for estimating passage quality allow passage corpora to be pruned considerably while maintaining statistically equivalent effectiveness; our best methods can consistently prune >25% of passages in a corpora, across various retrieval pipelines. Such substantial pruning reduces the operating costs of neural search engines in terms of computing resources, power usage, and carbon footprint -- both when processing queries (thanks to a smaller index size) and when indexing (lightweight models can prune low-quality passages prior to the costly dense or learned sparse encoding step). This work sets the stage for developing more advanced neural "learning-what-to-index" methods.
BibTeX
@misc{chang2024neural,
  author = {Xuejun Chang and Debabrata Mishra and Craig Macdonald and Sean MacAvaney},
  title = {{Neural Passage Quality Estimation for Static Pruning}},
  year = {2024},
  eprint = {2407.12170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.12170},
}
Figure