Abstract
Modern neural Information Retrieval (IR) models frequently prune low-frequency terms from their vocabularies to stay within memory limits. This paper investigates how such pruning affects the performance of state-of-the-art ranking models like KNRM, CONV-KNRM, and MatchPyramid. Our findings reveal that the negative impact of vocabulary pruning is closely tied to the re-ranking threshold—the number of documents processed by the neural model. We further demonstrate that subword-based embeddings, such as FastText, offer a superior solution, enhancing robustness on rare terms while maintaining a smaller memory footprint than full-vocabulary word-level models.
Introduction
In classical Information Retrieval, rare words are often the most informative, a principle formalized by Inverse Document Frequency (IDF). However, neural IR models typically utilize fixed-size embedding matrices. To accommodate GPU memory constraints, researchers often prune the vocabulary, removing terms that appear infrequently during training. This practice introduces two primary challenges: Out-Of-Vocabulary (OOV) terms at test time and poorly learned representations for rare words that remain. This study explores the trade-offs between vocabulary size, model architecture, and retrieval effectiveness.
Methodology
The authors evaluate three neural ranking architectures:
- KNRM: A kernel-based neural ranking model.
- CONV-KNRM: An extension of KNRM using convolutional layers to capture local context.
- MatchPyramid: A model that views text matching as image recognition on a matching matrix.
Two main strategies for handling vocabulary are compared:
- Vocabulary Pruning: Creating restricted vocabularies (e.g., Voc-5, Voc-50) by excluding terms with collection frequencies below a certain threshold.
- Subword Embeddings: Utilizing FastText, which represents words as bags of character n-grams. This allows the model to synthesize embeddings for OOV terms based on morphological similarity.
Experiments were conducted on the MS MARCO passage ranking dataset, involving 8.8 million documents and real-world Bing search queries.
The Impact of Re-ranking Thresholds
A critical discovery of this work is the "hidden" sensitivity of neural models to the re-ranking threshold. In a typical two-stage pipeline, a first-stage ranker (like BM25) retrieves a set of candidate documents, which the neural model then re-ranks.
As shown in Figure 1, models with smaller vocabularies are significantly more sensitive to the number of documents re-ranked. While performance initially improves as the threshold increases, it eventually peaks and then drops—often precipitously for pruned vocabularies. This suggests that as the model encounters more documents containing rare, poorly represented terms, the resulting "noise" degrades the overall ranking quality.
The FastText Advantage
FastText addresses the limitations of fixed vocabularies by providing a mechanism for generalization. By leveraging character n-grams, it can infer the semantics of rare or unseen words from their constituent parts.
The results in Figure 2 highlight that FastText is particularly effective for "tail" queries—those containing terms with very low collection frequencies (< 20). In these scenarios, traditional neural models often underperform even the classical BM25 baseline, whereas FastText remains robust.
Key Results and Efficiency
The study demonstrates that effectiveness does not have to come at the cost of efficiency:
- Performance: FastText achieved higher MRR@10 scores than the "Full Vocabulary" (Voc-Full) model across all architectures. For instance, with CONV-KNRM, FastText reached 0.275 MRR vs. 0.273 for Voc-Full.
- Memory Footprint: FastText is remarkably memory-efficient. While the Voc-Full model required 4.23 GB of memory, the FastText implementation achieved better results using only 2.36 GB (approximately 55% of the memory).
- Robustness: Tuning the re-ranking threshold is vital. For MatchPyramid with a Voc-50 vocabulary, optimal tuning resulted in an MRR of ~0.22, compared to ~0.17 at a higher, non-optimal threshold.
Conclusion
This research underscores the importance of low-frequency terms in neural IR. While vocabulary pruning is a common necessity, its detrimental effects can be mitigated through careful threshold tuning or, more effectively, by adopting subword-based embedding strategies like FastText. Future work may involve developing dynamic re-ranking strategies based on query performance prediction to further enhance robustness in the presence of rare terms.
Summary updated