Beyond Binary Relevance in Information Retrieval

282 concepts194 papers3 proposals

In traditional information retrieval (IR) and retrieval-augmented generation (RAG) systems, relevance is typically treated as a binary switch: a document is either relevant (1) or irrelevant (0). While this simplification makes it easy to compute classic evaluation metrics and train basic classifiers, it fails to capture the reality of how information satisfies a user's needs.

Information does not exist in black and white; it lives in a grayscale. A document might contain a perfect, comprehensive answer, or it might only offer a partial clue. Conversely, a document might be highly topically related but contain outdated or factually incorrect information. Forcing these nuanced relationships into a binary constraint creates an information bottleneck, limiting the precision of modern search engines, dense retrievers, and generative models.

The Limitations of the Binary Assumption

When a system is restricted to binary relevance labels, it suffers from two primary issues: information loss and noise injection.

As noted by Yongqi Li et al., binary categorization ignores the nuanced spectrum of relevance, treating moderately related or unlabeled documents with the same penalty as completely irrelevant ones. This loss of granularity prevents ranking models from performing the fine-grained comparisons necessary to order the top results accurately. Georgios Katsimpras et al. similarly observe that filtering documents via binary judgments discards critical information regarding the specific degree of relevance, which is vital for capturing subtle differences in document quality.

This limitation becomes particularly problematic in complex domains:

  • Factuality vs. Topicality: In RAG systems, a document can be topically relevant but factually incorrect. Giwon Hong et al. point out that binary classification fails to account for counterfactual noise or outdated information, leading models to trust and generate false outputs simply because a document matches the query's topic.
  • Legal Reasoning: In legal search, relevance exists on a complex spectrum of overlapping facts and statutory violations. T. Y. S. S. Santosh et al. argue that coarse-grained binary signals are fundamentally inadequate for high-stakes judicial reasoning where precedents carry varying degrees of authority.
  • Dialogue Systems: Dialogue models trained on binary labels struggle to navigate the vast space of intermediate-quality responses. As Deng Cai et al. demonstrate, a model must be able to distinguish between a perfect human response, a plausible but mediocre distractor, and complete random noise.

Quantifying the Grayscale: Ordinal and Soft Relevance

To overcome the binary bottleneck, researchers model relevance using more granular representational strategies. These generally fall into two categories: multi-level ordinal scales and continuous soft scoring.

Multi-Level Ordinal Scales

Instead of a 0/1 mapping, ordinal scales introduce a discrete hierarchy of utility. For instance, in legal contract drafting, Shanshan Wang et al. utilize an expert-annotated spectrum that explicitly distinguishes between a merely "relevant" clause and an "exemplary" one. Similarly, in video event detection, Zhongwen Xu et al. define multiple ordinal levels to incorporate "related exemplars"—videos that partially match an event definition—without polluting the training set with false positives.

Continuous and Soft Scoring

For complex, multi-faceted queries, relevance can be measured continuously by decomposing the user's request. The authors of Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability propose a soft scoring mechanism where a document's relevance is calculated as the exact proportion of sub-queries or decomposed claims it satisfies. This approach provides a direct mathematical representation of partial relevance, which is particularly useful when an answer must be synthesized from multiple distinct sources, as highlighted by Jingwei Ni et al.

Binary Relevance:
[Query] ---> [Document] ---> (Relevant OR Irrelevant)

Graded/Ordinal Relevance:
[Query] ---> [Document] ---> (Irrelevant | Partially Relevant | Highly Relevant | Exemplary)

Soft Relevance (Sub-query Coverage):
[Query] ---> Decompose into: {Sub-Q1, Sub-Q2, Sub-Q3} 
        ---> [Document] satisfies 2/3 ---> Relevance Score: 0.66

Designing Objectives for Graded Alignment

Moving to a graded relevance paradigm requires rewriting the loss functions used to train retrieval models. Standard contrastive learning objectives pull positive items closer and push negative items away with equal force. In a graded setting, the optimization objective must respect the relative ordering of the entire spectrum.

For example, in generative retrieval—where a model generates document identifiers directly—traditional architectures treat document targets as binary classes. To fix this, Yubao Tang et al. introduced a multi-graded constrained contrastive loss. This objective forces the model's internal representations to preserve the fine-grained distances between highly relevant, partially relevant, and irrelevant documents.

Another powerful mechanism is knowledge distillation. Instead of training a retriever on coarse human labels, the fine-grained, continuous ranking scores of an expensive cross-encoder model can be distilled into a faster student model. Yongqi Li et al. achieve this via a distilled RankNet loss, which teaches a generative student to mirror the nuanced, continuous probability distributions of the teacher, bypassing the limitations of discrete human annotations.

As Yisong Yue et al. demonstrated early on, algorithms that assume binary labels cannot natively exploit the rich structure of multi-level quality scores. By shifting to listwise or pairwise objectives that optimize for the relative order of items, models learn a much stronger supervisory signal.

Impact on Evaluation Metrics

If the underlying nature of relevance is graded, then evaluating systems using binary metrics like standard Average Precision (AP) or Recall@K becomes misleading. As Hao Li et al. point out, binary metrics fail to reflect how well a model captures broader semantic context or near-misses.

When evaluation frameworks adopt graded relevance scales, they rely on metrics like Normalized Discounted Cumulative Gain (NDCG) and Q-measure. As detailed in the Overview of the NTCIR-7 ACLIA IR4QA Task by Tetsuya Sakai et al., binary metrics can be overly punitive toward retrieval systems, failing to credit them for surfacing partially informative documents. Graded metrics ensure that a system is rewarded proportionally to the utility it delivers, aligning automated evaluation with actual human satisfaction.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_binary_relevance_limitation_7d5f7329,
  title = {binary relevance limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/binary-relevance-limitation/rdr_7d5f732933f0440c913e02f6db9f4baf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_binary_relevance_limitation_7d5f7329_v1,
  title = {binary relevance limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/binary-relevance-limitation/rdr_7d5f732933f0440c913e02f6db9f4baf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake