Traditional information retrieval (IR) and retrieval-augmented generation (RAG) systems operate on a binary assumption: a document is either relevant to a query or it is not. This "all-or-nothing" approach treats a perfect factual match the same as a tangentially related snippet, and a total mismatch the same as a "hard negative" that is topically similar but factually useless.
Recent research across 103 papers suggests that this binary bottleneck is a primary cause of model "gullibility," hallucinations, and poor ranking precision. By shifting toward Graded Relevance, researchers are teaching models to perceive the "grayscale" of information—the nuanced spectrum of utility that exists between a perfect answer and total noise.
The Binary Bottleneck
In a standard contrastive learning setup, a model is trained to pull "positive" pairs together and push "negative" pairs apart. However, as Peiyang Liu and colleagues argue in Improving Embedding-based Large-scale Retrieval via Label Enhancement, this discrete approach discards the continuous nature of semantic similarity.
When a model only sees 0s and 1s, it lacks the signal to distinguish between a "near-miss" and a "total mismatch." This is particularly dangerous in RAG systems. As Giwon Hong notes in Why So Gullible?, if a document is topically relevant but contains outdated or counterfactual information, a binary retriever will still label it as "relevant," leading the generator to produce false outputs.
Extracting the Spectrum
If the world isn't binary, where do we get the "grayscale" labels? Researchers have converged on three primary methods to generate fine-grained relevance signals:
- Teacher Distillation: Large, computationally expensive "Cross-Encoders" can look at a query and a document simultaneously to produce a precise relevance score. Yongqi Li and others in Distillation Enhanced Generative Retrieval use these scores to teach smaller, faster "Bi-Encoder" models to rank like their more powerful teachers.
- LLM-as-a-Judge: Large Language Models (LLMs) can be prompted to categorize documents into multiple levels (e.g., "Highly Relevant," "Partially Relevant," "Irrelevant"). Reza Esfandiarpoor demonstrates in Beyond Contrastive Learning that synthetic data with four distinct levels of relevance can outperform real-world binary annotations.
- Self-Guided Labeling: Models can learn to label their own data based on utility. In ARL2, Lingxi Zhang and team use an LLM to identify which documents actually help it answer a question, creating an "adaptive relevance" signal that prioritizes utility over surface-level similarity.
Training on the Gradient
Once we have multi-level labels, we need loss functions that can respect the hierarchy. Standard binary cross-entropy won't work because it doesn't know that a "Level 2" relevance is better than "Level 1."
- Ladder and Listwise Losses: Mo Zhou introduced Ladder Loss to ensure that even "incorrect" results are ranked sensibly—for example, ensuring a search for a "Golden Retriever" ranks a "Labrador" higher than a "Car." Similarly, Caili Guo uses a differentiable Smooth-NDCG loss to optimize the entire ranking list at once.
- Wasserstein Distance: In the SyCL framework, researchers use Wasserstein distance to measure the "cost" of misranking. This forces the model to pay a higher penalty for confusing a "Highly Relevant" document with an "Irrelevant" one than for confusing it with a "Partially Relevant" one.
- Graded Generative Retrieval: Even in the new frontier of Generative Retrieval—where models generate Document IDs directly—the spectrum matters. Yubao Tang and Wei Chen proposed GR$^2$, which uses a multi-graded constrained contrastive loss to ensure the model's internal probability distribution reflects the fine-grained relevance of the documents it is generating.
From Similarity to Utility
The ultimate goal of this shift is to move from semantic similarity (does this look like the query?) to information utility (does this help solve the task?).
In specialized domains like law, Yixiao Ma and the CaseEncoder team show that binary labels fail to capture the complex, weighted relationships between legal precedents. Similarly, in REAR, Yuhao Wang 0007 and Ruiyang Ren use a bi-granularity training strategy to give LLMs "self-awareness" of document relevance, allowing them to adaptively decide whether to trust retrieved content or rely on their own internal knowledge.
By embracing the spectrum of relevance, we move closer to systems that don't just find documents that "match," but find information that truly matters.