Toward the Understanding of Deep Text Matching Models for Information Retrieval

Lijuan Chen, Yanyan Lan, Liang Pang, Jiafeng Guo, Xueqi ChengarXiv 2021 · arXiv

Abstract

Deep learning models have revolutionized Information Retrieval (IR), yet they are often viewed as "black boxes" with opaque decision-making processes. This study investigates whether popular neural matching models—such as BERT, MatchPyramid, and KNRM—adhere to established IR heuristics like term frequency and length normalization. By testing these models against both classical and newly proposed semantic heuristics, and utilizing attribution analysis via Integrated Gradients, the research reveals that deep models are not merely guessing. Instead, they naturally align with and extend fundamental IR principles, demonstrating a sophisticated grasp of semantic relevance that surpasses traditional keyword matching.

Introduction

In recent years, deep learning models have dominated IR benchmarks like MS MARCO and LETOR. Despite their success, the "interpretability gap" persists: it is unclear if these models follow the logical constraints (heuristics) that have guided IR for decades. For instance, should a document with more query terms always rank higher? Traditional IR functions like BM25 are built on such mathematical certainties. This paper seeks to determine if complex, non-linear neural networks satisfy these "common sense" rules.

Methodology

The authors employ a two-pronged strategy to analyze model behavior:

Empirical Constraint Testing

The researchers construct specific query-document pairs designed to trigger IR heuristics. For example, to test the Term Frequency Constraint (TFC), they compare a document to a version where query terms are more frequent. If the model's score increases, the heuristic is satisfied.

Attribution Analysis

To understand why a model makes a specific ranking decision, the authors use Integrated Gradients (IG). This technique assigns an importance score to each input word, visualizing which terms contribute most to the final relevance score.

Integrated Gradients visualization for semantic matching

Semantic IR Heuristics

While classical heuristics rely on exact matches, the authors introduce Semantic Heuristics. These update traditional rules to account for word embeddings, allowing a match between "muscle" and "muscular" to be treated as a partial satisfaction of IR constraints.

Core Heuristics Evaluated

  1. Term Frequency (TFC): Higher query term frequency should generally increase scores, albeit with diminishing returns.
  2. Term Discrimination (TDC): Rare, high-IDF terms should carry more weight than common stop words.
  3. Length Normalization (LNC): Adding non-relevant "junk" words should decrease a document's score, while duplication should not penalize it.
  4. TF-LNC: Increasing query terms should improve scores even if the total document length increases.

Experimental Results

The study utilized the LETOR 4.0 and MS MARCO datasets to evaluate models including ARC-I (representation-based) and MatchPyramid/KNRM (interaction-based), alongside BERT.

Heuristic Satisfaction

Neural models demonstrated high adherence to classical heuristics, typically between 70% and 95%. Interaction-based models generally outperformed representation-based models in exact-match scenarios. However, satisfaction rates improved significantly when applying semantic versions of the rules, confirming that deep learning's primary strength lies in its ability to generalize relevance across the vector space.

Performance Benchmarks

On the MS MARCO dataset, BERT showed superior performance in terms of MRR and NDCG, while also satisfying the Term Discrimination Constraint (TDC) at a rate of 85.25%.

Model performance comparison on MS MARCO

Discussion and Limitations

While deep models are largely "principled," they occasionally struggle with length normalization. Attribution analysis revealed that "junk" words can sometimes carry unintended semantic weight that inflates a document's score. Furthermore, while BERT excels on large datasets like MS MARCO, it is prone to overfitting on smaller datasets like LETOR, which can impact its heuristic consistency.

Conclusion

This research provides evidence that deep IR models are not arbitrary; they effectively rediscover and apply the fundamental logic of information retrieval. Future work should focus on expanding the scope of heuristics to include document structure and using these constraints to guide the architectural design of more robust and interpretable IR systems.

Summary updated

Abstract

Semantic text matching is a critical problem in information retrieval. Recently, deep learning techniques have been widely used in this area and obtained significant performance improvements. However, most models are black boxes and it is hard to understand what happened in the matching process, due to the poor interpretability of deep learning. This paper aims at tackling this problem. The key idea is to test whether existing deep text matching methods satisfy some fundamental heuristics in information retrieval. Specifically, four heuristics are used in our study, i.e., term frequency constraint, term discrimination constraint, length normalization constraints, and TF-length constraint. Since deep matching models usually contain many parameters, it is difficult to conduct a theoretical study for these complicated functions. In this paper, We propose an empirical testing method. Specifically, We first construct some queries and documents to make them satisfy the assumption in a constraint, and then test to which extend a deep text matching model trained on the original dataset satisfies the corresponding constraint. Besides, a famous attribution based interpretation method, namely integrated gradient, is adopted to conduct detailed analysis and guide for feasible improvement. Experimental results on LETOR 4.0 and MS Marco show that all the investigated deep text matching methods, both representation and interaction based methods, satisfy the above constraints with high probabilities in statistics. We further extend these constraints to the semantic settings, which are shown to be better satisfied for all the deep text matching models. These empirical findings give clear understandings on why deep text matching models usually perform well in information retrieval. We believe the proposed evaluation methodology will be useful for testing future deep text matching models.
BibTeX
@misc{chen2021toward,
  author = {Lijuan Chen and Yanyan Lan and Liang Pang and Jiafeng Guo and Xueqi Cheng},
  title = {{Toward the Understanding of Deep Text Matching Models for Information Retrieval}},
  year = {2021},
  eprint = {2108.07081},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.07081},
}
Figure