Learning Term Discrimination

Jibril Frej, Philippe Mulhem, Didier Schwab, Jean-Pierre Chevallet

Abstract

Document indexing is a key component for efficient information retrieval (IR). Traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms traditional approaches in both nDCG and recall, even with limited training data. Furthermore, learned TDVs allow for significant index pruning, reducing memory footprint and speeding up retrieval by up to 3x without degrading quality.

Introduction

Information Retrieval (IR) has traditionally faced a trade-off between the efficiency of classical models and the effectiveness of modern neural approaches. Classical models like BM25 are highly efficient due to their use of inverted indexes, but they rely on static, heuristic formulas like Inverse Document Frequency (IDF) to weight term importance. Conversely, neural IR models offer superior accuracy but are often too computationally expensive for large-scale indexing.

This research bridges the gap by making classical IR functions differentiable. By learning Term Discrimination Values (TDVs) through a neural network, the system can optimize term weights for specific datasets while maintaining the speed and structure of an inverted index.

Methodology: Differentiable Term Weighting

The core of the proposed approach is a three-step strategy to integrate neural learning into classical indexing:

  1. Differentiable IR Functions: Standard IR formulas often rely on the non-differentiable $\ell_0$-norm. The authors replace this with an $\ell_1$-norm approximation, creating differentiable versions of TF-IDF, BM25, and Language Models.
  2. Shallow Neural TDV Learner: A simple neural network—consisting of a single linear layer and a ReLU activation—processes pre-trained word embeddings (e.g., fastText) to output a learned TDV for each vocabulary term.
  3. Supervised Pruning: The use of ReLU allows the model to output exactly zero for non-discriminative terms. This effectively performs "learned stop-word removal," where useless terms are pruned from the inverted index entirely.
The architecture of the proposed TDV learning framework, illustrating the forward and backward passes for optimizing term weights.

The model is trained using a pairwise hinge loss, optimizing the network to rank relevant document-query pairs higher than irrelevant ones.

Experimental Results

The approach was evaluated on three TREC collections: AP88-89, FT91-94, and LA. These datasets represent "small-data" scenarios, testing the model's efficiency when millions of training examples are unavailable.

Effectiveness

The learned TDVs consistently improved ranking quality across all datasets and base models. As shown in the results table, the TDV-enhanced versions of TF-IDF, LM, and BM25 outperformed their respective baselines in both nDCG@5 and Recall@1000.

Retrieval performance comparison showing that learned TDVs significantly improve nDCG@5 and Recall@1000 across multiple datasets.

Efficiency and Index Pruning

Beyond accuracy, the method provides substantial gains in system performance:

  • Retrieval Speed: On the AP88-89 dataset, TDV-BM25 achieved a 3x speedup compared to the standard baseline.
  • Index Size: By zeroing out non-discriminative terms, the memory footprint of the inverted index was reduced by 32% to 47% across the collections.

Discussion and Conclusion

This work demonstrates that complex, deep architectures are not always necessary to improve IR. A shallow network that "fixes" the weights of classical formulas can outperform heavier models like DRMM or DUET, especially when training data is scarce. By treating efficiency as a first-class citizen, the authors provide a practical path for deploying neural-enhanced retrieval in production environments.

Future work includes exploring the correlation between learned TDVs and traditional count-based IDF to better understand the model's logic, as well as testing the scalability of this pruning strategy on massive web-scale datasets.

Summary updated

Abstract

Document indexing is a key component for efficient information retrieval (IR). After preprocessing steps such as stemming and stop-word removal, document indexes usually store term-frequencies (tf). Along with tf (that only reflects the importance of a term in a document), traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms, both in terms of nDCG and recall, traditional approaches, even with few positively labelled query-document pairs as learning data. Our learned TDVs, when used to filter out terms of the vocabulary that have zero discrimination value, allow to both significantly lower the memory footprint of the inverted index and speed up the retrieval process (BM25 is up to 3~times faster), without degrading retrieval quality.
BibTeX
@inproceedings{frej2020learning,
  author = {Jibril Frej and Philippe Mulhem and Didier Schwab and Jean-Pierre Chevallet},
  title = {{Learning Term Discrimination}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  url = {https://openreview.net/forum?id=VkcKFNKNpx},
}
Figure