TL;DR
MIRA is a new evaluation benchmark for multi-category integrated retrieval, built from real user logs of a large-scale social science search platform. By leveraging LLMs to generate category-aware topic descriptions and scale graded relevance judgments across four distinct resource types (Publications, Datasets, Variables, and Tools), MIRA provides a realistic testbed for evaluating modern, unified search systems.
What Problem Does This Paper Address?
Traditional Information Retrieval (IR) evaluation has long relied on monolithic benchmarks (like classic TREC tracks) that focus on a single type of document, such as web pages or text passages. However, modern search systems—especially in scholarly and professional domains—increasingly require an integrated approach.
When a researcher enters a query, their information need is often multi-faceted: they might simultaneously look for research papers, the underlying datasets, specific survey variables, and specialized methodological tools. Existing benchmarks fail to reflect this multi-categorical reality, creating a gap between simplified evaluation environments and complex, real-world search interfaces. MIRA addresses this gap by providing a unified framework to evaluate category-aware ranking across heterogeneous data types.
Methods
The authors developed a highly automated, LLM-assisted pipeline to construct the benchmark from real-world data:
- Log-driven Topic Extraction: The authors analyzed over 16 million user interactions from the GESIS Search platform between 2017 and 2024, extracting query-item pairs based on explicit actions like viewing a record, downloading, or exporting citations.
- Query Clustering: To eliminate redundancy and group variant queries, they applied BERTopic using multilingual MiniLM embeddings, UMAP for dimensionality reduction, and HDBSCAN for clustering. They retained the 50 most frequent topical clusters for deep analysis.
- LLM-Assisted Curation: They utilized OpenAI's
gpt-5-minito generate structured, category-specific descriptions and narratives for 200 selected topics (145 in German, 55 in English). - Automated Relevance Judgments: Rather than relying purely on noisy implicit signals (like clicks), they used the LLM to provide explicit, graded relevance judgments (on a 0–4 scale) across a pooled set of 85,158 candidate documents.
- Expert Validation: To ensure data quality, four human experts independently annotated 10% of the dataset. The agreement between the human experts and the LLM achieved a quadratic-weighted Cohen’s kappa of $\kappa = 0.86$, indicating substantial agreement and validating the LLM's role as a reliable annotation assistant.
Datasets
The paper introduces the MIRA (Multi-Category Integrated Retrieval Assessment) dataset, which is a snapshot of four core scientific resource types from GESIS Search:
- Publications: 254,097 documents.
- Research Data: 7,634 quantitative social science datasets.
- Variables: 206,434 high-quality survey variables extracted from questionnaires.
- Instruments & Tools: 604 methodological resources (e.g., validated response scales, questionnaires).
In total, the collection contains metadata for 468,769 documents provided as JSON files, paired with 85,158 LLM-annotated relevance judgments across 200 topics.
Key Insights and Ideas
- The Multi-Categorical Challenge: Retrieval performance varies drastically depending on the target category. Ranking functions optimized for standard text documents (Publications) do not seamlessly transfer to highly structured metadata like survey variables or datasets, emphasizing the need for category-aware modeling.
- Implicit Signals are Noisy: The authors confirmed that user interactions (clicks/downloads) are imperfect proxies for topical relevance. A hybrid approach that uses interaction logs for candidate pooling and LLMs for explicit verification yields a much cleaner evaluation standard.
- LLMs as Scalable Judges: The high correlation ($\kappa = 0.86$) between human experts and the LLM demonstrates that LLMs can effectively scale the creation of complex, domain-specific IR benchmarks, drastically reducing the manual labor and cost traditionally required.
Key Results
The authors established baseline performance metrics using traditional lexical models (BM25, RLM) and state-of-the-art neural models (ColBERT, MonoT5) evaluated in a category-aware setting:
- Neural Superiority: Neural models (ColBERT and MonoT5) significantly outperformed lexical baselines across all categories and metrics ($p < 0.0001$).
- Model Strengths: ColBERT demonstrated highly consistent superiority, particularly dominating in the Publications (MAP: 0.5434) and Instruments & Tools (MAP: 0.4807) categories. Meanwhile, MonoT5 excelled at handling highly structured data, leading in Research Data (MAP: 0.4337) and Variables (MAP: 0.5249).
- Room for Improvement: Across all models, the highest Mean Average Precision (MAP) score remained below 0.6, confirming that multi-categorical retrieval in a scholarly domain remains a significant, unsolved challenge.
Limitations and Future Work
- Implicit Signal Noise: Although candidate pooling incorporated BM25 and ColBERT to mitigate exposure bias, the initial pool relied heavily on user interaction logs, which inherently contain behavioral noise.
- Future Scaling: The authors plan to expand MIRA by increasing both the number of judged topics and the depth of the relevance pools.
- Continuous Evaluation: They aim to leverage MIRA's automated workflow to create a "living evaluation corpus" that continuously updates as new user logs, topics, and documents evolve on the live search platform.
- Community Engagement: The team intends to organize a shared task or benchmark challenge to encourage the broader IR community to develop advanced cross-category and diversified ranking strategies.
Summary updated