IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval

Tingyu Song, Guo Gan, Mingsheng Shang, Yilun Zhao

Abstract: We introduce IFIR, the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. Each subset addresses one or more domain-specific retrieval tasks, replicating real-world scenarios where customized instructions are critical. IFIR enables a detailed analysis of instruction-following retrieval capabilities by incorporating instructions at different levels of complexity. We also propose a novel LLM-based evaluation method to provide a more precise and reliable assessment of model performance in following instructions. Through extensive experiments on 15 frontier retrieval models, including those based on LLMs, our results reveal that current models face significant challenges in effectively following complex, domain-specific instructions. We further provide in-depth analyses to highlight these limitations, offering valuable insights to guide future advancements in retriever development.

Introduction

Traditional Information Retrieval (IR) systems are typically evaluated on simple keyword queries or short natural language sentences. However, in "expert domains" such as legal research, financial analysis, or clinical medicine, user needs are far more nuanced. For example, a lawyer might need to find cases similar to a specific precedent, but only those where the plaintiff prevailed and damages were minimal.

Comparison of traditional multi-step retrieval versus end-to-end instruction-following retrieval in a legal context.

Current retrievers often ignore these complex constraints, forcing experts to manually filter through thousands of results. This gap highlights a critical need for both data and metrics to evaluate and improve end-to-end instruction-following in specialized fields. To address this, we present IFIR, a benchmark designed to replicate real-world expert demands.

Methodology

The construction of the IFIR benchmark follows a semi-automated, human-in-the-loop pipeline designed to ensure both scale and precision.

The three-step pipeline for constructing the IFIR benchmark: task selection, instruction generation, and passage annotation with human verification.

The methodology consists of three primary components:

  1. Instruction Generation: We utilized GPT-4o to generate instructions based on real-world expert demands, categorized into three levels of complexity (Simple, Context-rich, and Highly customized goals).
  2. Expert Validation: Domain experts (lawyers, doctors, etc.) reviewed and refined these instructions to ensure they were realistic and precise.
  3. INSTFOL Metric: To isolate instruction-following ability from general retrieval prowess, we proposed INSTFOL (Instruction-Following score). This LLM-based metric compares a retriever's performance with instructions against its performance without them, measuring the specific "instruction-following delta."

Dataset Overview

The IFIR benchmark consists of 2,426 high-quality examples across eight subsets derived from established IR datasets:

  • Finance: FiQA
  • Science: SciFact-open, NFCorpus
  • Law: AILA, FIRE
  • Healthcare: TREC-PM, TREC-CDS

Unlike original datasets, IFIR augments queries with domain-specific constraints that reflect actual professional workflows.

Key Insights and Analysis

Our benchmarking of 15 frontier retrievers revealed several significant findings:

  • The Instruction-Tuning Gap: Surprisingly, some instruction-tuned models did not significantly outperform their non-tuned base models on complex tasks. This suggests current instruction-tuning may overfit to simple, short instructions and fail to generalize to long-form expert constraints.
  • Lexical Bias in BM25: The classic BM25 algorithm performed relatively well. Expert instructions often contain specific glossary terms that act as strong anchors for traditional keyword search, even if the model doesn't fully "understand" the instruction's logic.
  • Complexity vs. Performance: As instructions move from Level 1 to Level 3 complexity, almost all models experience a performance dip. However, LLM-based retrievers (like GritLM-7B) showed greater robustness, suggesting that the reasoning capabilities of the underlying LLM are crucial for parsing complex constraints.
  • Long-Context Failures: In domains like law, instructions can exceed 1,000 tokens. Most current retrievers are capped at 512 tokens, causing them to lose critical information when faced with detailed case summaries.

Results and Conclusion

Among the tested models, NV-Embed-v2 and GritLM-7B emerged as leaders in overall retrieval accuracy (nDCG), while Promptriever-7B excelled in the INSTFOL metric, likely due to its targeted training on instance-level instructions. The new INSTFOL metric showed a high Pearson correlation (0.704) with human expert evaluations, proving it to be a reliable automated proxy for human judgment.

While IFIR marks a significant step forward, limitations remain regarding dataset scale and the potential for "false negatives" in relevance labels inherited from seed datasets. Future work will focus on creating larger-scale training sets to bridge the instruction-following gap in expert domains.

Summary updated

Abstract

We introduce IFIR, the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. Each subset addresses one or more domain-specific retrieval tasks, replicating real-world scenarios where customized instructions are critical. IFIR enables a detailed analysis of instruction-following retrieval capabilities by incorporating instructions at different levels of complexity. We also propose a novel LLM-based evaluation method to provide a more precise and reliable assessment of model performance in following instructions. Through extensive experiments on 15 frontier retrieval models, including those based on LLMs, our results reveal that current models face significant challenges in effectively following complex, domain-specific instructions. We further provide in-depth analyses to highlight these limitations, offering valuable insights to guide future advancements in retriever development.
BibTeX
@inproceedings{song2025ifir,
  author = {Tingyu Song and Guo Gan and Mingsheng Shang and Yilun Zhao},
  title = {{IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=3dRj0iU0de},
}
Figure