Abstract
With the recent addition of Retrieval-Augmented Generation (RAG), the scope and importance of Information Retrieval (IR) has expanded. As a result, the importance of a deeper understanding of IR models also increases. However, interpretability in IR remains under-explored, especially when it comes to the models' inner mechanisms. In this paper, we explore the possibility of adapting Integrated Gradient-based methods in an IR context to identify the role of individual neurons within the model. In particular, we provide new insights into the role of what we call "relevance" neurons, as well as how they deal with unseen data. Finally, we carry out an in-depth pruning study to validate our findings.
Introduction
Neural Information Retrieval (IR) models, particularly Cross-Encoders like MonoBERT, have redefined state-of-the-art performance. Despite their success, these models remain "black boxes," with little understood about their internal decision-making processes. While prior research has focused on input token importance, the role of individual neurons—the model's internal machinery—remains largely unexplored. Understanding these mechanisms is essential for improving model architecture, debugging failures, and facilitating domain adaptation, particularly when moving from general web search to specialized domains like medicine or finance.
Methodology: Neuron Integrated Gradients
To identify critical neurons, we adapt a technique called Neuron Integrated Gradients (NIG). Unlike standard Integrated Gradients (IG), which attributes predictions to input features (words), NIG attributes them to internal neuron activations.
The process involves three key steps:
- Baseline Selection: Establishing a "neutral" starting point. We found that replacing all input tokens with
[PAD]tokens serves as the most effective baseline for IR tasks. - Path Integration: Calculating gradients as the input evolves from the baseline to the actual query-document pair. Neurons that activate in correlation with the relevance score increase are tagged as important.
- Aggregation: Summing "conductance" (importance) across all tokens for linear transformations within the Transformer, specifically focusing on keys, queries, values, and feed-forward layers.
Functional Segregation and Domain Specialization
Our analysis reveals a striking functional segregation within the model. Neurons responsible for identifying "relevant" documents are almost entirely distinct from those identifying "non-relevant" ones, forming two separate internal circuits. This is evidenced by the extremely low intersection percentage between positive and negative neuron sets across various datasets.
Furthermore, we identified an "Out-of-Domain (OOD) Signature." When the model processes data from unfamiliar domains (e.g., FiQA for finance or BioASQ for medical), it relies on a consistent set of "specialist" neurons. These neurons are consistently active across different OOD tasks but are significantly less critical for the training domain (MSMARCO).
Layer-wise analysis shows that these important neurons cluster in two main areas:
- Middle Layers (7-12): Likely responsible for "relevance matching" between the query and the document.
- Final Layers (22-23): Associated with the
[CLS]token pooling, where signals are concentrated to produce the final score.
Validation via Pruning
To validate the importance of these neurons, we conducted extensive ablation studies. The results were definitive:
- High Sensitivity: Pruning just 0.1% of the neurons identified as important for MSMARCO led to a catastrophic 42.92% drop in nDCG@10 on the MSMARCO dev set.
- Random vs. Targeted: In contrast, pruning the same number of neurons randomly resulted in a negligible 0.09% performance change.
- OOD Impact: Pruning neurons identified through OOD datasets significantly harmed performance on the LoTTE benchmark, confirming that these neurons are indeed the functional specialists for unfamiliar domains.
Conclusion and Future Work
This work pulls back the curtain on the internal mechanics of Cross-Encoders for IR. We demonstrate that relevance and non-relevance are handled by distinct neural circuits and identify a specialized neural signature for out-of-domain data. While this study focused on MonoBERT, future work will explore if these findings hold for generative models (like MonoT5) and attempt to reverse-engineer the specific semantic features these neuron clusters detect.
Summary updated