Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

Daniel Cohen, Bhaskar Mitra, Oleg Lesota, Navid Rekabsaz, Carsten Eickhoff

Abstract

In any ranking system, the retrieval model outputs a single score for a document based on its belief on how relevant it is to a given search query. While retrieval models have continued to improve with the introduction of increasingly complex architectures, few works have investigated a retrieval model's belief in the score beyond the scope of a single value. We argue that capturing the model's uncertainty with respect to its own scoring of a document is a critical aspect of retrieval that allows for greater use of current models across new document distributions, collections, or even improving effectiveness for down-stream tasks. In this paper, we address this problem via an efficient Bayesian framework for retrieval models which captures the model's belief in the relevance score through a stochastic process while adding only negligible computational overhead. We evaluate this belief via a ranking based calibration metric showing that our approximate Bayesian framework significantly improves a retrieval model's ranking effectiveness through a risk aware reranking as well as its confidence calibration. Lastly, we demonstrate that this additional uncertainty information is actionable and reliable on down-stream tasks represented via cutoff prediction.

Introduction

Modern neural information retrieval (IR) models, particularly those based on large Transformers like BERT, are incredibly effective at ranking documents. However, they typically treat relevance as a deterministic point estimate—a single score per document. This approach obfuscates critical information: is the model absolutely sure a document is relevant, or is it highly uncertain because the document is out-of-distribution?

Ideally, an IR system should gracefully convey when it is no longer confident in its rankings. While ensemble methods (training multiple models and comparing their outputs) can capture this uncertainty, they are computationally infeasible for massive modern architectures. This research addresses the challenge of extracting reliable, well-calibrated uncertainty estimates from deep retrieval models without incurring the massive computational overhead of traditional ensembles.

Methodology: Efficient Bayesian Retrieval

The authors propose an approximate Bayesian inference approach using Monte Carlo (MC) Dropout. Instead of running a document through an entire ensemble of models, they run it through the same model multiple times with dropout enabled, creating a stochastic distribution of scores.

To make this computationally viable for heavy models like BERT, they leverage a theoretical finding: only the last layer(s) of a neural network need to be Bayesian to effectively capture uncertainty. This allows for generating a distribution of scores with negligible overhead.

Score distributions generated via MC-Dropout across different relevance levels. The model's belief is captured as a distribution rather than a single point estimate.

Key methodological contributions include:

  • Loss Function Adaptation: The standard pairwise hinge loss used in IR is mathematically relaxed to facilitate a Gaussian interpretation, allowing MC-Dropout to be applied properly to ranking tasks.
  • Expected Ranking Calibration Error (ERCE): Standard calibration metrics (like ECE) fail for IR because scores are relative per query. ERCE is a novel metric that measures calibration specifically between pairs of relevant and non-relevant documents within the same query.
  • Risk-Aware Reranking: Using Conditional Value at Risk (CVaR) to evaluate the generated score distributions allows the system to rerank documents based on a user-defined risk tolerance, penalizing documents with high variance or downside risk.

Experimental Setup

The framework was evaluated using several key datasets:

  • MS MARCO (Passage): Used for general training and evaluation.
  • TREC 2019 Deep Learning Track (DLT): Used for fine-grained evaluation of uncertainty properties, risk-aware reranking, and cutoff prediction.
  • Robust04: Used as an out-of-distribution test set. Models were evaluated on this zero-shot (no fine-tuning) to see how uncertainty behaves when the model encounters unfamiliar data.

Analysis of Uncertainty Properties

A key insight of this work is that uncertainty correlates strongly with relevance. As shown in the analysis of different model architectures, models tend to be highly confident (low variance) about highly relevant documents. As relevance drops, the model's variance increases, and the score distribution approaches a normal distribution.

Relationship between mean relevance and uncertainty metrics (variance and skew) across Conv-KNRM, BERT L4, and BERT L2 architectures.

Furthermore, standard calibration metrics are often flawed for search because document scores are only meaningful relative to other documents for the same query. The proposed ERCE metric provides a much more accurate picture of whether a model's confidence matches its actual ranking accuracy.

Key Results

  • Performance Parity: The mean scores of the MC-Dropout models achieve parity with their deterministic baselines (e.g., BERT-L4 achieved an MRR of 0.308 in both versions on MS MARCO). Adding uncertainty doesn't degrade base accuracy.
  • Reranking Boost: Risk-aware reranking using CVaR improved nDCG@200 by 3-5% over deterministic baselines on the TREC 2019 DLT dataset.
  • Superior Calibration: The Bayesian models are roughly 30% better calibrated than deterministic models according to the ERCE metric.
  • Better Cutoff Prediction: When using uncertainty features (variance, skew, entropy) to predict where to truncate a ranked list, F1 scores improved by ~9% on TREC DLT and up to 15% on the out-of-distribution Robust04 dataset.
  • Negligible Overhead: Running 100 MC samples on the last layer adds only ~0.368 μs of compute time per document for a BERT-L4 model.

Discussion and Future Work

While Bayesian models do not inherently perform better on out-of-distribution data (mean performance dropped slightly on Robust04), their advantage lies in "knowing what they don't know." This uncertainty is actionable and can be leveraged via secondary mechanisms like CVaR reranking or cutoff prediction to realize performance gains.

Future work includes applying these efficient stochastic models to areas like fairness, diversity, and dialogue agents. There is also potential for using uncertainty to improve sample efficiency during training or as a signal for information-gathering agents in complex multi-module systems.

Summary updated

Abstract

In any ranking system, the retrieval model outputs a single score for a document based on its belief on how relevant it is to a given search query. While retrieval models have continued to improve with the introduction of increasingly complex architectures, few works have investigated a retrieval model's belief in the score beyond the scope of a single value. We argue that capturing the model's uncertainty with respect to its own scoring of a document is a critical aspect of retrieval that allows for greater use of current models across new document distributions, collections, or even improving effectiveness for down-stream tasks. In this paper, we address this problem via an efficient Bayesian framework for retrieval models which captures the model's belief in the relevance score through a stochastic process while adding only negligible computational overhead. We evaluate this belief via a ranking based calibration metric showing that our approximate Bayesian framework significantly improves a retrieval model's ranking effectiveness through a risk aware reranking as well as its confidence calibration. Lastly, we demonstrate that this additional uncertainty information is actionable and reliable on down-stream tasks represented via cutoff prediction.
BibTeX
@inproceedings{cohen2021not,
  author = {Daniel Cohen and Bhaskar Mitra and Oleg Lesota and Navid Rekabsaz and Carsten Eickhoff},
  title = {{Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
  url = {https://openreview.net/forum?id=q06PhYyW1HC},
}
Figure