Abstract: We introduce IFIR, the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. Each subset addresses one or more domain-specific retrieval tasks, replicating real-world scenarios where customized instructions are critical. IFIR enables a detailed analysis of instruction-following retrieval capabilities by incorporating instructions at different levels of complexity. We also propose a novel LLM-based evaluation method to provide a more precise and reliable assessment of model performance in following instructions. Through extensive experiments on 15 frontier retrieval models, including those based on LLMs, our results reveal that current models face significant challenges in effectively following complex, domain-specific instructions. We further provide in-depth analyses to highlight these limitations, offering valuable insights to guide future advancements in retriever development.
Introduction
Traditional Information Retrieval (IR) systems are typically evaluated on simple keyword queries or short natural language sentences. However, in "expert domains" such as legal research, financial analysis, or clinical medicine, user needs are far more nuanced. For example, a lawyer might need to find cases similar to a specific precedent, but only those where the plaintiff prevailed and damages were minimal.
Current retrievers often ignore these complex constraints, forcing experts to manually filter through thousands of results. This gap highlights a critical need for both data and metrics to evaluate and improve end-to-end instruction-following in specialized fields. To address this, we present IFIR, a benchmark designed to replicate real-world expert demands.
Methodology
The construction of the IFIR benchmark follows a semi-automated, human-in-the-loop pipeline designed to ensure both scale and precision.
The methodology consists of three primary components:
- Instruction Generation: We utilized GPT-4o to generate instructions based on real-world expert demands, categorized into three levels of complexity (Simple, Context-rich, and Highly customized goals).
- Expert Validation: Domain experts (lawyers, doctors, etc.) reviewed and refined these instructions to ensure they were realistic and precise.
- INSTFOL Metric: To isolate instruction-following ability from general retrieval prowess, we proposed INSTFOL (Instruction-Following score). This LLM-based metric compares a retriever's performance with instructions against its performance without them, measuring the specific "instruction-following delta."
Dataset Overview
The IFIR benchmark consists of 2,426 high-quality examples across eight subsets derived from established IR datasets:
- Finance: FiQA
- Science: SciFact-open, NFCorpus
- Law: AILA, FIRE
- Healthcare: TREC-PM, TREC-CDS
Unlike original datasets, IFIR augments queries with domain-specific constraints that reflect actual professional workflows.
Key Insights and Analysis
Our benchmarking of 15 frontier retrievers revealed several significant findings:
- The Instruction-Tuning Gap: Surprisingly, some instruction-tuned models did not significantly outperform their non-tuned base models on complex tasks. This suggests current instruction-tuning may overfit to simple, short instructions and fail to generalize to long-form expert constraints.
- Lexical Bias in BM25: The classic BM25 algorithm performed relatively well. Expert instructions often contain specific glossary terms that act as strong anchors for traditional keyword search, even if the model doesn't fully "understand" the instruction's logic.
- Complexity vs. Performance: As instructions move from Level 1 to Level 3 complexity, almost all models experience a performance dip. However, LLM-based retrievers (like GritLM-7B) showed greater robustness, suggesting that the reasoning capabilities of the underlying LLM are crucial for parsing complex constraints.
- Long-Context Failures: In domains like law, instructions can exceed 1,000 tokens. Most current retrievers are capped at 512 tokens, causing them to lose critical information when faced with detailed case summaries.
Results and Conclusion
Among the tested models, NV-Embed-v2 and GritLM-7B emerged as leaders in overall retrieval accuracy (nDCG), while Promptriever-7B excelled in the INSTFOL metric, likely due to its targeted training on instance-level instructions. The new INSTFOL metric showed a high Pearson correlation (0.704) with human expert evaluations, proving it to be a reliable automated proxy for human judgment.
While IFIR marks a significant step forward, limitations remain regarding dataset scale and the potential for "false negatives" in relevance labels inherited from seed datasets. Future work will focus on creating larger-scale training sets to bridge the instruction-following gap in expert domains.
Summary updated