Abstract
We introduce IFIR, the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. Each subset addresses one or more domain-specific retrieval tasks, replicating real-world scenarios where customized instructions are critical. IFIR enables a detailed analysis of instruction-following retrieval capabilities by incorporating instructions at different levels of complexity. We also propose a novel LLM-based evaluation method to provide a more precise and reliable assessment of model performance in following instructions. Through extensive experiments on 15 frontier retrieval models, including those based on LLMs, our results reveal that current models face significant challenges in effectively following complex, domain-specific instructions. We further provide in-depth analyses to highlight these limitations, offering valuable insights to guide future advancements in retriever development.
Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in following instructions within conversational contexts, Information Retrieval (IR) systems have lagged behind. Traditional IR benchmarks, such as BEIR or MTEB, primarily evaluate models based on short, keyword-centric queries. However, in expert domains like law, medicine, or finance, users rarely rely on keywords alone. A lawyer searching for case law or a clinician seeking specific clinical trials often provides detailed constraints, personal context, and nuanced goals.
Existing retrievers frequently ignore these detailed instructions, defaulting to simple pattern matching. This creates a bottleneck for expert users who must manually decompose complex needs into simplified queries. To address this, we present IFIR, a benchmark specifically designed to measure how well IR systems handle the dense, multi-layered instructions characteristic of expert fields.
The IFIR Benchmark
IFIR consists of 2,426 high-quality examples distributed across eight subsets and four specialized domains:
- Finance: Utilizing the FiQA dataset.
- Law: Utilizing AILA and FIRE datasets.
- Healthcare: Utilizing TREC-PM and TREC-CDS.
- Science Literature: Utilizing SciFact-open and NFCorpus.
Each example includes a query, a detailed instruction, and ground-truth relevant passages verified by domain experts. To create these instructions, we employed a semi-automated, human-in-the-loop pipeline. Using GPT-4o, we generated instructions at three distinct levels of complexity:
- Level 1 (Simple): Basic task descriptions.
- Level 2 (Contextual): Added personal context, such as specific patient demographics or business structures.
- Level 3 (Constrained): Highly specific goals and constraints, such as seeking tax efficiency for a specific legal entity.
Evaluation Methodology: The INSTFOL Metric
Standard metrics like nDCG@20 measure whether a retriever finds the correct document, but they fail to isolate whether the model actually followed the instruction or simply got lucky with keyword matching. To solve this, we introduced INSTFOL (Instruction-Following Score).
INSTFOL uses an LLM (GPT-4o-mini) to measure the relative improvement a retriever shows when instructions are provided versus when they are absent. By accounting for the "room for improvement," INSTFOL ensures that models are not penalized for high baseline performance on simple queries. Our validation shows that INSTFOL has a strong Pearson correlation of 0.704 with human expert evaluations, making it a robust automated proxy for instruction-following capability.
Key Insights and Experimental Results
We evaluated 15 frontier retrieval models, ranging from traditional bi-encoders to modern LLM-based retrievers. Several key findings emerged:
1. The Limits of Instruction-Tuning
Surprisingly, some models specifically marketed as "instruction-tuned" (e.g., INSTRUCTOR) showed minimal improvement over non-tuned counterparts like GTR. This suggests that current instruction-tuning datasets may be overfit to general-domain prompts, failing to generalize to the dense, logic-heavy instructions found in expert domains.
2. The "BM25 Surprise"
The classic BM25 algorithm remains a formidable baseline. Expert instructions are often rich in domain-specific terminology. If a neural model fails to parse the complex logic of an instruction, it can be outperformed by simple keyword matching that happens to catch the "glossary terms" embedded in the prompt.
3. The Complexity Penalty
Most models experienced a performance dip when moving from Level 1 to Level 2 instructions. The added context often acted as "noise" that distracted the models. However, at Level 3, performance occasionally improved, as the highly detailed constraints provided more explicit anchors for the models to utilize.
4. LLMs as the Path Forward
Retrievers built on LLM architectures, such as GritLM-7B and Promptriever-7B, significantly outperformed standard bi-encoders. Their ability to process long-context information allows them to "understand" constraints rather than treating instructions as a simple bag of words. NV-Embed-v2 and GritLM-7B led in raw performance (nDCG@20 of 0.51), while Promptriever-7B achieved the highest INSTFOL score (8.00%).
Conclusion and Future Work
IFIR reveals a sobering reality: even our most advanced retrievers struggle with the nuances of expert-domain instructions. While LLM-based models show promise, there is significant room for improvement. Future research should focus on:
- Scaling: Expanding the benchmark beyond 2,400 queries.
- Training Sets: Developing specialized training data to help models learn expert-domain instruction following.
- Context Windows: Addressing the 512-token limit of standard bi-encoders, which is often insufficient for complex legal or medical instructions.
- Domain Specialization: Comparing general-purpose models against those specifically fine-tuned for law or medicine.
Summary updated