On the Effect of Low-Frequency Terms on Neural-IR Models

Sebastian Hofstätter, Navid Rekabsaz, Carsten Eickhoff, Allan Hanbury

Abstract

Modern neural Information Retrieval (IR) models frequently prune low-frequency terms from their vocabularies to stay within memory limits. This paper investigates how such pruning affects the performance of state-of-the-art ranking models like KNRM, CONV-KNRM, and MatchPyramid. Our findings reveal that the negative impact of vocabulary pruning is closely tied to the re-ranking threshold—the number of documents processed by the neural model. We further demonstrate that subword-based embeddings, such as FastText, offer a superior solution, enhancing robustness on rare terms while maintaining a smaller memory footprint than full-vocabulary word-level models.

Introduction

In classical Information Retrieval, rare words are often the most informative, a principle formalized by Inverse Document Frequency (IDF). However, neural IR models typically utilize fixed-size embedding matrices. To accommodate GPU memory constraints, researchers often prune the vocabulary, removing terms that appear infrequently during training. This practice introduces two primary challenges: Out-Of-Vocabulary (OOV) terms at test time and poorly learned representations for rare words that remain. This study explores the trade-offs between vocabulary size, model architecture, and retrieval effectiveness.

Methodology

The authors evaluate three neural ranking architectures:

  1. KNRM: A kernel-based neural ranking model.
  2. CONV-KNRM: An extension of KNRM using convolutional layers to capture local context.
  3. MatchPyramid: A model that views text matching as image recognition on a matching matrix.

Two main strategies for handling vocabulary are compared:

  • Vocabulary Pruning: Creating restricted vocabularies (e.g., Voc-5, Voc-50) by excluding terms with collection frequencies below a certain threshold.
  • Subword Embeddings: Utilizing FastText, which represents words as bags of character n-grams. This allows the model to synthesize embeddings for OOV terms based on morphological similarity.

Experiments were conducted on the MS MARCO passage ranking dataset, involving 8.8 million documents and real-world Bing search queries.

The Impact of Re-ranking Thresholds

A critical discovery of this work is the "hidden" sensitivity of neural models to the re-ranking threshold. In a typical two-stage pipeline, a first-stage ranker (like BM25) retrieves a set of candidate documents, which the neural model then re-ranks.

Sensitivity of neural IR models to the re-ranking threshold across different vocabulary sizes. Smaller vocabularies (e.g., Voc-50) exhibit a sharp performance decline as more documents are re-ranked.

As shown in Figure 1, models with smaller vocabularies are significantly more sensitive to the number of documents re-ranked. While performance initially improves as the threshold increases, it eventually peaks and then drops—often precipitously for pruned vocabularies. This suggests that as the model encounters more documents containing rare, poorly represented terms, the resulting "noise" degrades the overall ranking quality.

The FastText Advantage

FastText addresses the limitations of fixed vocabularies by providing a mechanism for generalization. By leveraging character n-grams, it can infer the semantics of rare or unseen words from their constituent parts.

Heatmap of MRR@10 differences between FastText and the Full Vocabulary model. Red regions indicate where FastText outperforms the word-level model, particularly for queries containing very low-frequency terms.

The results in Figure 2 highlight that FastText is particularly effective for "tail" queries—those containing terms with very low collection frequencies (< 20). In these scenarios, traditional neural models often underperform even the classical BM25 baseline, whereas FastText remains robust.

Key Results and Efficiency

The study demonstrates that effectiveness does not have to come at the cost of efficiency:

  • Performance: FastText achieved higher MRR@10 scores than the "Full Vocabulary" (Voc-Full) model across all architectures. For instance, with CONV-KNRM, FastText reached 0.275 MRR vs. 0.273 for Voc-Full.
  • Memory Footprint: FastText is remarkably memory-efficient. While the Voc-Full model required 4.23 GB of memory, the FastText implementation achieved better results using only 2.36 GB (approximately 55% of the memory).
  • Robustness: Tuning the re-ranking threshold is vital. For MatchPyramid with a Voc-50 vocabulary, optimal tuning resulted in an MRR of ~0.22, compared to ~0.17 at a higher, non-optimal threshold.

Conclusion

This research underscores the importance of low-frequency terms in neural IR. While vocabulary pruning is a common necessity, its detrimental effects can be mitigated through careful threshold tuning or, more effectively, by adopting subword-based embedding strategies like FastText. Future work may involve developing dynamic re-ranking strategies based on query performance prediction to further enhance robustness in the presence of rare terms.

Summary updated

Abstract

Low-frequency terms are a recurring challenge for information retrieval models, especially neural IR frameworks struggle with adequately capturing infrequently observed words. While these terms are often removed from neural models - mainly as a concession to efficiency demands - they traditionally play an important role in the performance of IR models. In this paper, we analyze the effects of low-frequency terms on the performance and robustness of neural IR models. We conduct controlled experiments on three recent neural IR models, trained on a large-scale passage retrieval collection. We evaluate the neural IR models with various vocabulary sizes for their respective word embeddings, considering different levels of constraints on the available GPU memory. We observe that despite the significant benefits of using larger vocabularies, the performance gap between the vocabularies can be, to a great extent, mitigated by extensive tuning of a related parameter: the number of documents to re-rank. We further investigate the use of subword-token embedding models, and in particular FastText, for neural IR models. Our experiments show that using FastText brings slight improvements to the overall performance of the neural IR models in comparison to models trained on the full vocabulary, while the improvement becomes much more pronounced for queries containing low-frequency terms.
BibTeX
@inproceedings{hofstatter2019effect,
  author = {Sebastian Hofstätter and Navid Rekabsaz and Carsten Eickhoff and Allan Hanbury},
  title = {{On the Effect of Low-Frequency Terms on Neural-IR Models}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2019},
  doi = {10.1145/3331184.3331344},
}
Figure