Abstract
Document indexing is a key component for efficient information retrieval (IR). Traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms traditional approaches in both nDCG and recall, even with limited training data. Furthermore, learned TDVs allow for significant index pruning, reducing memory footprint and speeding up retrieval by up to 3x without degrading quality.
Introduction
Information Retrieval (IR) has traditionally faced a trade-off between the efficiency of classical models and the effectiveness of modern neural approaches. Classical models like BM25 are highly efficient due to their use of inverted indexes, but they rely on static, heuristic formulas like Inverse Document Frequency (IDF) to weight term importance. Conversely, neural IR models offer superior accuracy but are often too computationally expensive for large-scale indexing.
This research bridges the gap by making classical IR functions differentiable. By learning Term Discrimination Values (TDVs) through a neural network, the system can optimize term weights for specific datasets while maintaining the speed and structure of an inverted index.
Methodology: Differentiable Term Weighting
The core of the proposed approach is a three-step strategy to integrate neural learning into classical indexing:
- Differentiable IR Functions: Standard IR formulas often rely on the non-differentiable $\ell_0$-norm. The authors replace this with an $\ell_1$-norm approximation, creating differentiable versions of TF-IDF, BM25, and Language Models.
- Shallow Neural TDV Learner: A simple neural network—consisting of a single linear layer and a ReLU activation—processes pre-trained word embeddings (e.g., fastText) to output a learned TDV for each vocabulary term.
- Supervised Pruning: The use of ReLU allows the model to output exactly zero for non-discriminative terms. This effectively performs "learned stop-word removal," where useless terms are pruned from the inverted index entirely.
The model is trained using a pairwise hinge loss, optimizing the network to rank relevant document-query pairs higher than irrelevant ones.
Experimental Results
The approach was evaluated on three TREC collections: AP88-89, FT91-94, and LA. These datasets represent "small-data" scenarios, testing the model's efficiency when millions of training examples are unavailable.
Effectiveness
The learned TDVs consistently improved ranking quality across all datasets and base models. As shown in the results table, the TDV-enhanced versions of TF-IDF, LM, and BM25 outperformed their respective baselines in both nDCG@5 and Recall@1000.
Efficiency and Index Pruning
Beyond accuracy, the method provides substantial gains in system performance:
- Retrieval Speed: On the AP88-89 dataset, TDV-BM25 achieved a 3x speedup compared to the standard baseline.
- Index Size: By zeroing out non-discriminative terms, the memory footprint of the inverted index was reduced by 32% to 47% across the collections.
Discussion and Conclusion
This work demonstrates that complex, deep architectures are not always necessary to improve IR. A shallow network that "fixes" the weights of classical formulas can outperform heavier models like DRMM or DUET, especially when training data is scarce. By treating efficiency as a first-class citizen, the authors provide a practical path for deploying neural-enhanced retrieval in production environments.
Future work includes exploring the correlation between learned TDVs and traditional count-based IDF to better understand the model's logic, as well as testing the scalability of this pruning strategy on massive web-scale datasets.
Summary updated