Learning Term Discrimination

Jibril Frej, Philippe Mulhem, Didier Schwab, Jean-Pierre Chevallet

Abstract

Document indexing is a key component for efficient information retrieval (IR). After preprocessing steps such as stemming and stop-word removal, document indexes usually store term-frequencies (tf). Along with tf (that only reflects the importance of a term in a document), traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms, both in terms of nDCG and recall, traditional approaches, even with few positively labelled query-document pairs as learning data. Our learned TDVs, when used to filter out terms of the vocabulary that have zero discrimination value, allow to both significantly lower the memory footprint of the inverted index and speed up the retrieval process (BM25 is up to 3 times faster), without degrading retrieval quality.

Introduction

In traditional Information Retrieval (IR), document ranking relies on two primary components: Term Frequency (tf), which measures the local importance of a word within a document, and Inverse Document Frequency (idf), which measures the global discriminative power of a word across a collection.

Despite their success, traditional IR systems face two major hurdles. First, standard discrimination measures like idf are static and not optimized for specific ranking tasks. Second, the process of removing non-discriminative "stop-words" is typically a manual, binary decision that doesn't fully exploit the potential for index pruning. While deep neural networks can learn better representations, they are often too computationally expensive for large-scale production retrieval. This paper proposes a method to learn Term Discrimination Values (TDVs) using shallow neural networks, bridging the gap between the efficiency of traditional inverted indexes and the learning power of modern neural models.

Methodology: Differentiable IR Functions

The core innovation lies in redefining traditional ranking formulas—such as TF-IDF, BM25, and Dirichlet Language Models—to be mathematically differentiable. By replacing non-differentiable components (like the $\ell_0$-norm used in idf) with differentiable approximations (like the $\ell_1$-norm), the researchers enable the use of gradient descent to optimize the parameters of traditional models.

Architecture of the proposed differentiable TDV-TF-IDF model, showing the forward and backward passes for learning term discrimination values.

Shallow Neural Network and Joint Optimization

The model utilizes a simple neural network consisting of a single linear layer with a ReLU activation. This network takes pre-trained word embeddings as input and outputs a learned TDV for every term in the vocabulary. The training process involves a joint objective:

  1. Ranking Accuracy: A pairwise hinge loss is used to improve the model's ability to rank relevant documents higher than irrelevant ones.
  2. Sparsity Constraint: An $\ell_1$ regularization objective encourages the network to assign a TDV of exactly zero to non-discriminative terms.

Because of the ReLU activation, the model can output true zeros, effectively acting as an automated, data-driven stop-word generator. Any term assigned a TDV of zero is pruned from the inverted index, reducing its size and accelerating retrieval.

Experimental Results

The models were evaluated on three standard TREC collections: AP88-89 (Associated Press), FT91-94 (Financial Times), and LA (Los Angeles Times).

Ranking and Efficiency

The learned TDV versions of TF-IDF, BM25, and LM consistently outperformed their baseline counterparts in both nDCG@5 and Recall@1000. Beyond accuracy, the efficiency gains were substantial. By filtering out terms with zero discrimination value, the retrieval process became significantly faster. For instance, on the AP88-89 collection, BM25 retrieval was over 3x faster than the baseline.

Comparison of retrieval time (ms) between baseline models and their TDV-optimized counterparts across three TREC collections.

Index Pruning and Memory Footprint

The sparsity objective successfully reduced the memory footprint of the inverted index by 32% to 47% across different collections without degrading retrieval quality. Furthermore, these shallow models proved to be dramatically faster than complex neural baselines like DRMM or Conv-KNRM, which are often too slow for initial large-scale retrieval stages.

Conclusion

This research demonstrates that we do not have to choose between the interpretability and speed of traditional IR models and the optimization capabilities of neural networks. By making traditional functions differentiable and learning term discrimination values, we can achieve better ranking accuracy while simultaneously shrinking the index and speeding up retrieval. Future work will focus on scaling these methods to larger datasets like MS MARCO and exploring the semantic nuances captured by learned TDVs compared to traditional count-based formulas.

Summary updated

Abstract

Document indexing is a key component for efficient information retrieval (IR). After preprocessing steps such as stemming and stop-word removal, document indexes usually store term-frequencies (tf). Along with tf (that only reflects the importance of a term in a document), traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms, both in terms of nDCG and recall, traditional approaches, even with few positively labelled query-document pairs as learning data. Our learned TDVs, when used to filter out terms of the vocabulary that have zero discrimination value, allow to both significantly lower the memory footprint of the inverted index and speed up the retrieval process (BM25 is up to 3~times faster), without degrading retrieval quality.
BibTeX
@inproceedings{frej2020learning,
  author = {Jibril Frej and Philippe Mulhem and Didier Schwab and Jean-Pierre Chevallet},
  title = {{Learning Term Discrimination}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  url = {https://openreview.net/forum?id=VkcKFNKNpx},
}
Figure