When you search a massive digital library or the web for a specific fact—such as a technical threshold, a legal precedent, or a precise experimental result—traditional search engines often present you with a list of long-form documents. You click a link, only to find yourself staring at a 50-page PDF or a massive forum thread, forced to manually scan thousands of words to find a single relevant sentence.
This friction stems from a fundamental architectural problem in information retrieval: the retrieval granularity mismatch. While human information needs are often highly specific and fine-grained, traditional retrieval systems index and score information at a coarse, document-centric level.
The Core Dilemma of Document-Centric Scoring
To understand why this mismatch occurs, we have to look at how traditional retrieval models calculate relevance. Standard algorithms rely heavily on document-wide statistics, such as term frequencies or global vector embeddings.
When relevance is computed across an entire document, localized information gets diluted. As noted by Qingyao Ai et al. in UMass at TREC 2017 Common Core Track, document-wide statistics frequently obscure highly relevant specific segments within long or diverse texts. If a single paragraph contains the exact answer to a query but is surrounded by pages of unrelated text, the global score of the document drops. Consequently, the document may be ranked far below a shorter, globally thematic but less precise document, hiding the exact answer from the user.
Decomposing the Document Structure
The earliest attempts to resolve this mismatch focused on changing the unit of indexing itself. If document-level retrieval is too coarse, why not index the internal structural components of the document?
For instance, Yosi Mass et al. in Retrieving the most relevant XML Components adapted the Vector Space Model to index individual document elements, such as paragraphs and sections. By isolating these fragments, the system could bypass document-level dilution and point users directly to the relevant sub-component.
However, completely isolating text fragments introduces a new problem: the loss of context. A single paragraph might contain the target keyword but become incomprehensible without the surrounding chapter's context. To balance this trade-off, Börkur Sigurbjörnsson et al. in An Element-based Approach to XML Retrieval developed language models that dynamically mix evidence from both the individual element and its parent document. This approach preserves localized precision while leveraging global context to validate the element's true relevance.
The Two-Stage Retrieve-and-Extract Paradigm
Instead of completely re-engineering the search index, a parallel school of thought addresses the granularity mismatch by splitting the task into two sequential phases: broad document retrieval followed by localized answer extraction.
Qi Li formalized this strategy in Searching for Entities When Retrieval Meets Extraction through the Two-layer Retrieval and Extraction Probability Model (TREPM). The intuition here is pragmatic: standard search engines are exceptionally good at finding globally relevant "supporting documents." Once the search space is narrowed down to a few high-quality documents, specialized extraction models can take over to isolate discrete entities, tables, or text spans.
This pipeline directly bridges the "answer extraction gap" identified in early question-answering systems. As argued by Laurie Gerber et al. in Question Answering in Webclopedia, while traditional information retrieval effectively identifies relevant articles, it fails to extract the concise facts required by natural language queries. A two-stage architecture allows each component to operate at its optimal granularity.
Autoregressive Substring Generation
With the rise of deep learning, researchers have pioneered a more radical solution that eliminates the distinction between retrieval and extraction entirely. Instead of retrieving a document pointer and then scanning it, what if a model could directly generate the exact text span containing the answer?
A prominent example of this paradigm shift is the SEAL framework, introduced by Michele Bevilacqua et al. in Autoregressive Search Engines: Generating Substrings as Document Identifiers. Rather than mapping a query to a static document ID or a dense vector representation, SEAL trains an autoregressive language model to generate any valid substring (n-gram) directly from the corpus as a unique identifier.
By constraining the model's generation to an internal FM-Index of the corpus, the system guarantees that the output corresponds to an exact, verifiable text span. This design allows the system to achieve extreme granularity, pinpointing precise evidence directly through generation and bypassing the limitations of coarse-grained vector matching.
Impact on Generation and Verifiability
Resolving the granularity mismatch is not just about improving search engine results; it is a critical requirement for making Retrieval-Augmented Generation (RAG) systems trustworthy. When large language models (LLMs) are fed entire, noisy documents as context, they are prone to missing nuanced details or hallucinating facts.
To counter this, the IRP framework developed by Nishant Balepur et al. in Expository Text Generation: Imitate, Retrieve, Paraphrase shifts the generation process to an iterative, sentence-level loop. By generating fine-grained content plans and using those plans as highly specific search queries, the system retrieves precise text fragments rather than broad topics, significantly increasing factual density.
[Coarse RAG] Query ──> Document-Level Retrieval ──> Large Context (with Noise) ──> Potential Hallucination
[Fine-Grained] Query ──> Sentence-Level Planning ──> Exact Substring Retrieval ──> Verifiable Generation
Furthermore, this precision transforms how humans interact with AI-generated content. Document-level citations force users to manually audit extensive texts to verify a single claim. To eliminate this bottleneck, Suifeng Zhao et al. introduced the AWeCita framework in AWeCita: Generating Answer with Appropriate and Well-grained Citations Using LLMs. By training models to cite exact text spans rather than monolithic files, the system aligns the granularity of the evidence with the granularity of the generated claim, drastically reducing the cognitive load required for human verification.