TL;DR
ProSearch is a new benchmark designed to evaluate how well AI systems can retrieve scientific protocols that satisfy multiple, interdependent constraints (e.g., "find a 3D culture for C2C12 cells using alginate"). It reveals that current state-of-the-art retrievers struggle significantly with "constraint binding"—the ability to ensure all conditions are met within a single experimental configuration—highlighting a major bottleneck for AI in experimental science.
What Problem Does This Paper Address?
In the burgeoning field of "AI for Science," researchers need systems that can design experiments and reproduce methods. This requires more than just finding "relevant" papers; it requires finding specific protocols that meet a precise set of requirements.
Traditional Information Retrieval (IR) benchmarks focus on topical relevance. However, scientific method paragraphs often describe multiple experimental variants in one place. A standard keyword-based search might find a paragraph that mentions "cell line X," "37°C," and "DMEM," but those conditions might actually belong to three different experiments described in that same text. The challenge is joint constraint binding: verifying that all query conditions hold true within the same coherent experimental setup. Existing benchmarks lack the complexity and diagnostic depth to measure this specific capability.
Methods
The authors developed an expert-guided construction pipeline to ensure the benchmark is both realistic and verifiable.
- Expert Modeling: Domain experts defined 60 experimental fields (like
cell_type,temperature,solvent) and their dependencies. - Structured Extraction: Using advanced LLMs (specifically GPT-5.2 in the paper's timeline), the authors extracted structured records from a pool of scientific method paragraphs. Each record represents one coherent experimental configuration.
- Query Generation: Instead of arbitrary keyword combinations, queries were grounded in these real experimental records. They used templates (e.g., Goal + Input + Condition) to generate 1,515 natural language queries.
- Relevance Labeling: To avoid the pitfalls of simple keyword matching, the authors used a "Near-Miss" candidate mining strategy. They identified paragraphs that matched all but one constraint to find challenging examples. Final relevance was judged by an LLM (GPT-5.4) and audited by human experts to ensure that a paragraph is only "relevant" if a single setup satisfies every constraint.
Datasets
The paper introduces ProSearch, which consists of:
- 1,515 queries across six domains: Cell Culture, Immunotechnology, Electrochemistry, Organic Synthesis, Inorganic Materials, and Polymer Materials.
- 3–5 explicit constraints per query, including numerical ranges and negations (e.g., "excluding DMEM").
- A retrieval corpus of method paragraphs extracted from open-access articles (2020–2025).
The authors compare ProSearch to existing benchmarks like BRIGHT and SciFact, noting that ProSearch is unique in requiring "Hard" satisfaction of all requirements and "Binding" of those requirements to the same context.
Key Insights and Ideas
- The "Constraint Binding" Bottleneck: Manual error analysis showed that 90% of top-ranked non-relevant results failed because they violated at least one constraint. Models are good at finding the right "ingredients" but bad at checking if they are in the same "recipe."
- Generalists Outperform Specialists: Interestingly, general-purpose dense retrievers (like Qwen3-Embedding) outperformed models specifically trained on scientific text. This suggests that the reasoning required for multi-constraint matching benefits more from broad training data than from narrow domain adaptation.
- Negation is a Major Hurdle: Queries involving negation (e.g., "medium other than RPMI") showed significantly lower performance, indicating that current embedding-based retrievers struggle to process "not" in a scientific context.
- Structured Data is the "Gold Standard": A diagnostic experiment showed that if you use the extracted structured records for retrieval, performance jumps from 0.290 to 0.814 nDCG@10. This proves the information is there; the retrievers just can't "see" the structure in the raw text.
Key Results
The evaluation of 19 representative retrievers paints a sobering picture of the current state of the art:
- Low Baseline Performance: The best-performing first-stage retriever (Qwen3-Embedding-8B) achieved an average nDCG@10 of only 0.290.
- Reranking is Essential: Applying a reranker (like Qwen3-Reranker) nearly doubled the score to 0.474, though this still leaves significant room for improvement.
- Domain Variance: Performance was particularly low in complex domains like Immunotechnology and Electrochemistry, where experimental descriptions are more nuanced.
- Expert Validation: Human experts agreed with the benchmark's relevance labels in 89.1% of cases, confirming the high quality of the dataset.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope: The benchmark currently covers six domains and focuses on single-paragraph retrieval. Real-world scientific evidence often spans multiple paragraphs, tables, and supplementary materials.
- Template Bias: While grounded in real data, the queries are generated via templates and may not capture the full variety of how researchers naturally phrase questions.
- LLM Dependency: The pipeline relies heavily on LLMs for extraction and labeling, which could introduce systematic biases.
Future Work: The team plans to extend ProSearch to cross-paragraph retrieval and incorporate queries authored directly by researchers to better simulate real-world scientific discovery workflows.
Summary updated