MessIRve: A Large-Scale Spanish Information Retrieval Dataset

Francisco Valentini, Viviana Cotik, Damián Furman, Ivan Bercovich, Edgar Altszyler, Juan Manuel PérezEMNLP 2025

TL;DR

This paper introduces MessIRve, a large-scale Spanish information retrieval (IR) dataset designed to address the scarcity of resources for Spanish IR research. By leveraging Google's autocomplete API and Wikipedia, MessIRve offers a diverse collection of natural, non-translated queries that capture dialectal variations across Spanish-speaking countries, aiming to advance IR systems for this significant language group.

What Problem Does This Paper Address?

Information retrieval (IR) systems are crucial for finding relevant information, and their performance heavily relies on comprehensive evaluation datasets. While English has a wealth of such resources, Spanish, the second most spoken native language globally, suffers from a significant lack of dedicated IR datasets. Existing Spanish IR datasets are often limited by their size, reliance on translated queries (which can introduce errors and reduce relevance), restricted topical coverage, or a lack of dialectal diversity. This gap hinders the development and evaluation of effective IR systems tailored for the diverse needs of Spanish speakers.

Methods

The authors employed an automated data collection strategy inspired by the GooAQ dataset. They used Google's autocomplete API to generate queries, starting with a predefined set of Spanish question words (e.g., "qué," "cómo," "dónde"). To find relevant documents, they appended "wikipedia" to these queries and extracted the featured snippets from Google Search results, using the associated Wikipedia paragraphs as documents. This approach aimed to capture natural user queries and relevant, real-world content. The dataset was then split into training and test sets, ensuring that Wikipedia articles were partitioned between splits to minimize topic overlap. A quality assessment was conducted by independent annotators and LLMs to validate the dataset's correctness, unambiguity, and document relevance.

Datasets

The primary contribution of this paper is the introduction of MessIRve, a new large-scale Spanish IR dataset. It comprises approximately 700,000 queries sourced from Google's autocomplete API and relevant documents extracted from Wikipedia. The dataset is designed to be dialectally diverse, covering queries from 20 Spanish-speaking countries and the United States.

The authors also compare MessIRve to existing Spanish IR datasets, including:

  • mMARCO: A translated version of the MSMARCO dataset.
  • SQAC: A Spanish Question Answering Corpus.
  • MIRACL: A multilingual IR dataset including Spanish.
  • PRES: A Spanish Passage Retrieval dataset focused on health.
  • Multi-EuP: A dataset using European Parliament debates.

Key Insights and Ideas

  • The Power of Natural Queries: Using Google's autocomplete API, rather than human annotation or translation, yields queries that are more representative of real user search behavior, including colloquialisms and potential grammatical imperfections.
  • Dialectal Diversity is Crucial: MessIRve's explicit inclusion of queries from various Spanish-speaking countries highlights the significant dialectal variations in language use, which are often overlooked in other datasets. This is crucial for building truly inclusive IR systems.
  • Wikipedia as a Rich Source: Leveraging Wikipedia as the document corpus provides a vast and diverse range of topics, making MessIRve suitable for training IR models that can handle a wide array of information needs.
  • Automated Collection Balances Scale and Cost: The automated approach allows for the creation of a large-scale dataset efficiently, overcoming the high cost and time associated with manual annotation, though it introduces potential biases from the source (Google).

Key Results

  • Dataset Size and Diversity: MessIRve is significantly larger than most existing Spanish IR datasets, with over 700,000 queries and a broad topical coverage derived from over 84,000 unique Wikipedia articles.
  • Query Characteristics: MessIRve queries are generally shorter and less variable in length compared to other datasets, reflecting typical search engine query patterns. They also exhibit higher prefix diversity, indicating more varied question structures.
  • Baseline Model Performance: In experiments, dense retrieval models (OpenAI-large, E5-large, MIRACL-mdpr-es) consistently outperformed the lexical BM25 model on the MessIRve test set. The fine-tuned E5-large model achieved the highest nDCG@10 scores, while zero-shot OpenAI-large led in Recall@100.
  • Quality Assessment: Manual evaluation by independent raters and LLMs indicated high quality, with over 90% agreement on query correctness and document relevance, and substantial agreement on unambiguity.

Limitations and Future Work

  • Representativeness: The dataset may not be fully representative of all popular topics in each country, as it captures a snapshot of popular queries at a specific time. It also only includes queries for which Google is confident in providing an answer on Wikipedia.
  • Potential Biases: Reliance on Google's "featured snippets" may introduce biases reflecting Google's ranking priorities and content preferences, potentially leading to systems that outperform Google's own technology.
  • Dialectal Underrepresentation: Despite efforts, certain dialects or linguistic variations might still be underrepresented.
  • Future Work: The authors suggest future work including fine-tuning models on country-specific subsets, creating a unified Spanish IR benchmark with all datasets, and further investigating the impact of dialectal variations. They also acknowledge the need for more comprehensive annotations, including negative judgments, to further improve IR system training.

Summary updated

BibTeX
@inproceedings{valentini2025messirve,
  author = {Francisco Valentini and Viviana Cotik and Damián Furman and Ivan Bercovich and Edgar Altszyler and Juan Manuel Pérez},
  title = {{MessIRve: A Large-Scale Spanish Information Retrieval Dataset}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure