Abstract
This paper studies the consistency of the kernel-based neural ranking model K-NRM, a recent state-of-the-art neural IR model, which is important for reproducible research and deployment in the industry. We find that K-NRM has low variance on relevance-based metrics across experimental trials. In spite of this low variance in overall performance, different trials produce different document rankings for individual queries. The main source of variance in our experiments was found to be different latent matching patterns captured by K-NRM. In the IR-customized word embeddings learned by K-NRM, the query-document word pairs follow two different matching patterns that are equally effective, but align word pairs differently in the embedding space. The different latent matching patterns enable a simple yet effective approach to construct ensemble rankers, which improve K-NRM's effectiveness and generalization abilities.
Introduction
In Information Retrieval (IR), neural models have become state-of-the-art due to their ability to handle "soft matches"—understanding semantic relationships (e.g., "sofa" vs. "couch") beyond exact keyword matching. However, these models are non-convex and trained stochastically, leading to potential inconsistencies across training runs. For industrial deployment and reproducible research, understanding whether the same architecture on the same data yields consistent results is critical. This study investigates the stability of the Kernel Neural Ranking Model (K-NRM), looking beyond aggregate scores to individual query-level ranking decisions.
Methodology and Datasets
The researchers conducted 50 independent trials of K-NRM with random initializations using a massive dataset from Sogou.com click logs containing 95 million queries. Stability was analyzed through aggregate performance metrics (like NDCG), query-level agreement, and internal parameter analysis (kernel weights and word embeddings).
Testing was performed on 1,000 queries with 30 candidate documents each, evaluated against three label types to ensure robustness: Testing-SAME (same click model as training), Testing-DIFF (sophisticated click model), and Testing-RAW (actual user clicks).
Latent Matching Patterns and Stability
A primary discovery is that K-NRM trials converge into one of two distinct "latent matching patterns." These patterns represent different internal strategies for prioritizing soft-matches. Remarkably, both patterns are equally effective in terms of aggregate accuracy, despite their different internal logic.
The Illusion of Stability
While aggregate performance metrics show low standard deviation (0.5–1.3%), the actual rankings for individual queries vary significantly. In a set of 10 trials, only 5% of queries resulted in all 10 models agreeing on the top-ranked document.
As shown in the figure above, for the majority of queries, the union of top-ranked documents across 10 runs is much greater than one, indicating substantial disagreement between trials despite similar overall scores.
Results: Ensemble Performance and Generalization
The diversity in latent patterns provides an opportunity for ensemble learning. By combining trials that exhibit different patterns, researchers can leverage the "wisdom of the crowd" to improve performance and generalization.
- Ensemble Gains: An ensemble mixing both Pattern A and Pattern B (Ensemble-A&B) achieved a 14% to 21% relative improvement in NDCG@1 over individual models.
- Generalization: Ensembles performed better on
Testing-DIFFandTesting-RAWsets, indicating that combining different internal "viewpoints" helps the model adapt to unseen data.
This variance scales as more documents are considered; the union of top-10 documents across trials often exceeds 20 or 30 unique documents, highlighting the diverse document spaces explored by different training runs.
Conclusion and Future Work
K-NRM is consistent in aggregate but varies at the query level due to distinct latent matching patterns. This variation is not a flaw but a feature that can be exploited via ensembles to boost ranking effectiveness and generalization. Future work should investigate if these patterns persist in more complex architectures like BERT-based rankers and explore methods to encourage this diversity within single training runs to reduce computational costs.
Summary updated