Consistency and Variation in Kernel Neural Ranking Model

Mary Arpita Pyreddy, Varshini Ramaseshan, Narendra Nath Joshi, Zhuyun Dai, Chenyan Xiong, Jamie Callan, Zhiyuan Liu

TL;DR

While the Kernel Neural Ranking Model (K-NRM) produces consistent overall performance across different training runs, it often reaches that performance through two distinct, equally effective "latent matching patterns." By identifying these different internal strategies, researchers can build ensembles that significantly outperform individual models and improve ranking stability.

What Problem Does This Paper Address?

In the world of Information Retrieval (IR), neural models have become the state-of-the-art due to their ability to handle "soft matches" (understanding that "sofa" and "couch" are related) rather than just exact keyword matching. However, these models are "non-convex" and trained stochastically, meaning every time you train the model, it might end up in a slightly different state.

For industrial deployment and reproducible research, this raises a red flag: Is the model consistent? If two researchers train the same architecture on the same data, will they get the same results? This paper investigates the stability of K-NRM, looking beyond aggregate scores to see if the model actually makes the same ranking decisions for individual queries across different trials.

Methods

The authors conducted a rigorous stability study by training 50 independent trials of the K-NRM model with random initializations. They analyzed the model from three angles:

  1. Aggregate Performance: Comparing standard metrics like NDCG (Normalized Discounted Cumulative Gain) across runs.
  2. Query-Level Agreement: Checking if different trials agree on which document should be ranked #1 for a specific query.
  3. Internal Parameter Analysis: Examining the weights assigned to the "kernels" (which group word-pair similarities into bins) and the resulting word embeddings.

To capitalize on their findings, they also tested ensemble models—combining multiple trained versions of K-NRM to see if "wisdom of the crowd" could mitigate individual trial variances.

Datasets

The study utilized a massive real-world dataset:

  • Sogou.com Click Logs: A Chinese web search engine dataset containing 95 million queries for training.
  • Testing Sets: 1,000 queries with 30 candidate documents each.
  • Labels: The authors used three types of labels to ensure robustness: Testing-SAME (using the same click model as training), Testing-DIFF (using a more sophisticated click model), and Testing-RAW (using actual user clicks as the ground truth).

Key Insights and Ideas

  • The "Two Patterns" Discovery: This is the paper's "aha moment." The authors found that K-NRM trials don't just vary randomly; they tend to converge into one of two distinct "latent matching patterns." One pattern might prioritize certain types of soft-matches while the other prioritizes different ones.
  • Equally Effective Paths: Surprisingly, both Pattern A and Pattern B produce nearly identical aggregate accuracy. There isn't a "better" way to learn; there are just two different ways the model organizes its internal logic.
  • Stability is an Illusion: While the average score of the model is very stable (low standard deviation), the actual rankings for individual queries vary wildly. In 10 trials, only 5% of queries had all 10 models agree on the top document.
  • Diversity Drives Ensembles: Because the two patterns "disagree" on individual rankings but are both generally "smart," they are perfect candidates for an ensemble. A model that combines trials from both Pattern A and Pattern B performs better than an ensemble made of just one pattern.

Key Results

  • Consistency: The standard deviation for performance metrics was small (0.5–1.3%), suggesting K-NRM is reliable for research reporting.
  • Ensemble Gains: Ensembles significantly outperformed individual models. Specifically, the "Ensemble-A&B" (mixing both patterns) showed a 14% to 21% relative improvement in NDCG@1 over the average individual model.
  • Generalization: Ensembles showed better performance on the Testing-DIFF and Testing-RAW sets, suggesting that combining different "viewpoints" from different trials helps the model generalize better to new, unseen data.

Limitations and Future Work

  • Scope of Models: The study focuses exclusively on K-NRM. While K-NRM is a foundational interaction-based model, it remains to be seen if more complex architectures (like BERT-based rankers, which became dominant shortly after this paper) exhibit similar "dual-pattern" convergence.
  • Dataset Specificity: The results are based on a single (albeit very large) Chinese search dataset. Testing across different languages and domains (like medical or legal search) would confirm if these latent patterns are a universal property of kernel-based ranking.
  • Computational Cost: Training 50 models to find patterns for an ensemble is computationally expensive. Future work could explore how to encourage these diverse patterns within a single training run (e.g., through specific regularization or multi-head architectures).

Summary updated

Abstract

This paper studies the consistency of the kernel-based neural ranking model K-NRM, a recent state-of-the-art neural IR model, which is important for reproducible research and deployment in the industry. We find that K-NRM has low variance on relevance-based metrics across experimental trials. In spite of this low variance in overall performance, different trials produce different document rankings for individual queries. The main source of variance in our experiments was found to be different latent matching patterns captured by K-NRM. In the IR-customized word embeddings learned by K-NRM, the query-document word pairs follow two different matching patterns that are equally effective, but align word pairs differently in the embedding space. The different latent matching patterns enable a simple yet effective approach to construct ensemble rankers, which improve K-NRM's effectiveness and generalization abilities.
BibTeX
@inproceedings{pyreddy2018consistency,
  author = {Mary Arpita Pyreddy and Varshini Ramaseshan and Narendra Nath Joshi and Zhuyun Dai and Chenyan Xiong and Jamie Callan and Zhiyuan Liu},
  title = {{Consistency and Variation in Kernel Neural Ranking Model}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2018},
  url = {https://openreview.net/forum?id=HyNSCrZdZr},
}

Loading related content...

Figure