Abstract
Deep learning models have revolutionized Information Retrieval (IR), yet they are often viewed as "black boxes" with opaque decision-making processes. This study investigates whether popular neural matching models—such as BERT, MatchPyramid, and KNRM—adhere to established IR heuristics like term frequency and length normalization. By testing these models against both classical and newly proposed semantic heuristics, and utilizing attribution analysis via Integrated Gradients, the research reveals that deep models are not merely guessing. Instead, they naturally align with and extend fundamental IR principles, demonstrating a sophisticated grasp of semantic relevance that surpasses traditional keyword matching.
Introduction
In recent years, deep learning models have dominated IR benchmarks like MS MARCO and LETOR. Despite their success, the "interpretability gap" persists: it is unclear if these models follow the logical constraints (heuristics) that have guided IR for decades. For instance, should a document with more query terms always rank higher? Traditional IR functions like BM25 are built on such mathematical certainties. This paper seeks to determine if complex, non-linear neural networks satisfy these "common sense" rules.
Methodology
The authors employ a two-pronged strategy to analyze model behavior:
Empirical Constraint Testing
The researchers construct specific query-document pairs designed to trigger IR heuristics. For example, to test the Term Frequency Constraint (TFC), they compare a document to a version where query terms are more frequent. If the model's score increases, the heuristic is satisfied.
Attribution Analysis
To understand why a model makes a specific ranking decision, the authors use Integrated Gradients (IG). This technique assigns an importance score to each input word, visualizing which terms contribute most to the final relevance score.
Semantic IR Heuristics
While classical heuristics rely on exact matches, the authors introduce Semantic Heuristics. These update traditional rules to account for word embeddings, allowing a match between "muscle" and "muscular" to be treated as a partial satisfaction of IR constraints.
Core Heuristics Evaluated
- Term Frequency (TFC): Higher query term frequency should generally increase scores, albeit with diminishing returns.
- Term Discrimination (TDC): Rare, high-IDF terms should carry more weight than common stop words.
- Length Normalization (LNC): Adding non-relevant "junk" words should decrease a document's score, while duplication should not penalize it.
- TF-LNC: Increasing query terms should improve scores even if the total document length increases.
Experimental Results
The study utilized the LETOR 4.0 and MS MARCO datasets to evaluate models including ARC-I (representation-based) and MatchPyramid/KNRM (interaction-based), alongside BERT.
Heuristic Satisfaction
Neural models demonstrated high adherence to classical heuristics, typically between 70% and 95%. Interaction-based models generally outperformed representation-based models in exact-match scenarios. However, satisfaction rates improved significantly when applying semantic versions of the rules, confirming that deep learning's primary strength lies in its ability to generalize relevance across the vector space.
Performance Benchmarks
On the MS MARCO dataset, BERT showed superior performance in terms of MRR and NDCG, while also satisfying the Term Discrimination Constraint (TDC) at a rate of 85.25%.
Discussion and Limitations
While deep models are largely "principled," they occasionally struggle with length normalization. Attribution analysis revealed that "junk" words can sometimes carry unintended semantic weight that inflates a document's score. Furthermore, while BERT excels on large datasets like MS MARCO, it is prone to overfitting on smaller datasets like LETOR, which can impact its heuristic consistency.
Conclusion
This research provides evidence that deep IR models are not arbitrary; they effectively rediscover and apply the fundamental logic of information retrieval. Future work should focus on expanding the scope of heuristics to include document structure and using these constraints to guide the architectural design of more robust and interpretable IR systems.
Summary updated