ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge

Zeinab Taghavi, Ali Modarressi, Yunpu Ma, Hinrich Schuetze2025 · DOI

Introduction

In the current landscape of Retrieval-Augmented Generation (RAG), the prevailing assumption is that if a piece of information exists within a corpus, a retriever will successfully locate it. However, most existing benchmarks, such as BEIR, focus on "surface-level" cues—lexical overlap and semantic similarity. While recent research has pushed the boundaries of query complexity (e.g., multi-hop reasoning), the complexity of the documents themselves has been largely neglected.

This paper introduces IMPLIRET, a benchmark designed to shift the reasoning burden from the query to the document side. It reveals a significant gap in modern AI: even state-of-the-art retrievers and long-context Large Language Models (LLMs) struggle to connect the dots when facts are stated implicitly rather than explicitly.

The IMPLIRET Benchmark

IMPLIRET is a reasoning-intensive corpus where facts are never stated verbatim. The benchmark consists of 9,000 documents, structured so that while the queries are straightforward (e.g., "Who was in the UK?"), the relevance of a document is hidden behind a layer of implicit reasoning.

Data Generation Pipeline

The authors developed a rigorous four-stage pipeline to ensure the quality and "implicitness" of the data:

  1. Implicit-Tuple Generation: Facts were categorized into three reasoning types: Arithmetic (e.g., price comparisons), World Knowledge (e.g., linking landmarks like Big Ben to countries), and Temporal (e.g., relative dates like "three days ago").
  2. Discourse Styling: To simulate real-world data, documents were generated in two styles: Uni-speaker (multi-turn chats) and Multi-speaker (forum threads with multiple contributors).
  3. LLM-Based Synthesis: Gemma-3-27B was used to synthesize natural-sounding conversations based on these tuples.
  4. Verification Step: A second LLM was tasked with extracting the original implicit fact from the generated text. If the fact was not fully recoverable through inference, the document was discarded to maintain benchmark integrity.

Experimental Results

The researchers evaluated a diverse suite of retrieval systems, including sparse models (BM25), late-interaction models (ColBERT v2), dense encoders (Dragon+, Contriever), and Knowledge-Graph augmented systems (HippoRAG 2).

Retrieval is "Blind" to Inference

The results were sobering. The highest average nDCG@10 achieved was a mere 14.91% by Dragon+. For context, the traditional BM25 model trailed closely at 12.24%. This marginal difference suggests that modern dense retrievers are still largely performing sophisticated pattern matching rather than true logical inference during the encoding process. They are effectively "blind" to the underlying reasoning required to link a query to an implicit fact.

The "Lost in the Context" Problem

The study also explored an "Oracle" setting, where the correct document was guaranteed to be within the LLM's context window (k=30). Even with the correct information present, the performance of models like GPT-4o-mini plummeted to 55.54%.

When provided with only the positive document (k=1), LLMs like GPT-4o-mini and Llama 3.3 70B performed excellently, reaching 80-92% recall. However, the addition of 29 "hard negative" distractors—documents that are structurally similar but factually irrelevant—caused a performance drop of nearly 40%. This proves that a long context window is not a silver bullet; LLMs struggle to isolate correct evidence when it is buried among distracting, semantically similar information.

Key Insights and Error Analysis

  • Style Impacts Difficulty: Multi-speaker forum threads proved significantly more challenging than uni-speaker chats. The fragmented nature of information across different posts makes it harder for models to build a coherent representation of implicit facts.
  • Semantic Distractors: In arithmetic tasks, retrievers often fell for "Semantic Cues." They frequently selected documents that discussed the correct brands or products but contained the wrong price calculations, indicating a reliance on keyword proximity over mathematical logic.
  • Word Overlap: In temporal and world-knowledge tasks, simple word overlap remained the primary driver of misranking, leading retrievers to favor documents with shared vocabulary over those with logical relevance.

Limitations and Future Directions

While IMPLIRET provides a crucial new lens for evaluation, it has limitations. The documents are synthetic, and while verified for fluency, they may not capture the full linguistic noise of organic human communication. Additionally, the reasoning is currently "shallow," focusing on single-step inferences.

Future work should explore multi-step implicit reasoning (e.g., combining an arithmetic calculation with a temporal shift) and expand the benchmark to languages beyond English. Most importantly, the authors suggest a need for "reasoning-aware" indexing—systems that perform some level of inference before or during the document embedding process, rather than relying on post-hoc retrieval logic.

Summary updated

Abstract

Retrieval systems are central to many NLP pipelines, but often rely on surface-level cues such as keyword overlap and lexical semantic similarity.To evaluate retrieval beyond these shallow signals, recent benchmarks introduce reasoning-heavy queries; however, they primarily shift the burden to query-side processing techniques -like prompting or multi-hop retrieval -that can help resolve complexity.In contrast, we present IMPLIRET, a benchmark that shifts the reasoning challenge to documentside processing: The queries are simple, but relevance depends on facts stated implicitly in documents through temporal (e.g., resolving "two days ago"), arithmetic, and world knowledge relationships.We evaluate a range of sparse and dense retrievers, all of which struggle in this setting: the best nDCG@10 is only 14.91%.We also test whether long-context models can overcome this limitation.But even with a short context of only thirty documents, including the positive document, GPT-o4-mini scores only 55.54%, showing that document-side reasoning remains a challenge.Our codes are available at github.com/ZeinabTaghavi/IMPLIRET.
BibTeX
@misc{taghavi2025impliret,
  author = {Zeinab Taghavi and Ali Modarressi and Yunpu Ma and Hinrich Schuetze},
  title = {{ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1685},
}
Figure