Overview of INEX 2013

Patrice Bellot, Antoine Doucet, Shlomo Geva, Sairam Gurajada, Jaap Kamps, Gabriella Kazai, Marijn Koolen, Arunav Mishra, Véronique Moriceau, Josiane Mothe, Michael Preminger, Eric SanJuan, Ralf Schenkel, Xavier Tannier, Martin Theobald, Matthew Trappett, Qiuyue Wang

TL;DR

INEX 2013, a workshop focused on information retrieval, brought together researchers to tackle challenges in searching diverse and structured data. This paper summarizes the four tracks of the 2013 campaign: Social Book Search, Linked Data, Tweet Contextualization, and Snippet Retrieval, highlighting their aims, datasets, and initial findings, and emphasizing the creation of valuable test collections for future research.

What Problem Does This Paper Address?

Traditional information retrieval (IR) often treats documents as simple "bags of words." However, the modern web and professional search environments are increasingly reliant on structured data, including document structure, semantic metadata, entities, and genre. This shift presents a challenge: how can IR systems effectively search and retrieve information from these complex, multi-faceted sources? INEX (Information Needs and Exploration) has been at the forefront of addressing this by developing test collections and evaluation measures for structured retrieval since 2002. INEX 2013 continued this mission by organizing four tracks that explored different facets of "focused retrieval" – the idea of retrieving specific pieces of information rather than entire documents. This paper provides an overview of the INEX 2013 activities, aiming to foster collaboration and knowledge transfer between different evaluation forums, particularly by integrating with the CLEF (Conference and Labs of the Evaluation Forum) initiative.

Methods

INEX 2013 comprised four distinct tracks, each with its own specific aims and methodologies:

  • Social Book Search Track: This track focused on searching digitized books and associated metadata, including user-generated content. It aimed to explore techniques for understanding complex user needs beyond simple topical relevance, incorporating aspects like genre, recency, engagement, and quality. Two tasks were included: the Social Book Search task, simulating a user searching a book catalog, and the Prove It task, testing focused retrieval for factual claims within books.
  • Linked Data Track: This track investigated retrieval over structured and semantic data, specifically using DBpedia and Wikipedia. It aimed to bridge the gap between keyword-based IR and Semantic Web reasoning by exploring how structural information and RDF properties can enhance retrieval and enable navigation through large datasets. Two tasks were offered: an Ad Hoc Retrieval task for keyword queries and a Jeopardy task for natural language queries translated into SPARQL.
  • Tweet Contextualization Track: This track addressed the challenge of understanding tweets by providing a short, automatically generated summary using local resources like Wikipedia. The objective was to help users grasp the context of a tweet, moving beyond simple keyword matching. The task involved extracting relevant passages from Wikipedia and aggregating them into a coherent summary.
  • Snippet Retrieval Track: This track focused on generating informative snippets for search results. The goal was to create snippets that allow users to quickly determine a document's relevance without needing to view the entire document. Participants were tasked with returning a ranked list of documents along with a corresponding text snippet, with constraints on snippet length.

Datasets

INEX 2013 utilized and expanded upon several key datasets:

  • Social Book Search Track: A new test collection was developed based on real-world information needs from LibraryThing (LT) discussion forums and user profiles. This collection included 2.8 million book descriptions from Amazon, enriched with user reviews and LT user-generated content. Relevance judgments were derived from forum suggestions. The Prove It task used a collection of over 50,000 digitized out-of-copyright books (approximately 17 million pages) marked up in XML.
  • Linked Data Track: The track used a subset of DBpedia 3.8 and YAGO2s, along with a recent dump of Wikipedia core articles (June 2012). Supplementary collections were provided, including a fusion of XML'ified Wikipedia articles with RDF properties from DBpedia and YAGO2s, and a dump of textual Wikipedia content in RDF. A set of 144 Ad-hoc task search topics and 105 Jeopardy task search topics were released.
  • Tweet Contextualization Track: The document collection was based on a November 2011 dump of the English Wikipedia, filtered to include only non-empty pages with at least one section. In 2013, 120 tweets, manually collected and checked by organizers, were used as topics.
  • Snippet Retrieval Track: This track used the same collection as the Tweet Contextualization track: an XML version of the English Wikipedia from November 2012. A specific subset of 700 documents was provided for participants to generate snippets from. 35 topics were used, with 10 from INEX 2010 and 25 newly created for this track.

Key Insights and Ideas

  • The growing importance of structured data: The paper consistently highlights that pure text retrieval is no longer sufficient. The increasing prevalence of structured information on the web and in professional contexts necessitates new approaches to information access.
  • Bridging the gap between IR and Semantic Web: The Linked Data track explicitly aims to connect traditional keyword search with Semantic Web technologies, suggesting a future where these paradigms are more integrated.
  • The value of user-generated content: The Social Book Search track demonstrates the utility of incorporating user reviews, tags, and forum discussions to understand complex information needs and provide richer search results.
  • Focused retrieval is key: All tracks, in different ways, emphasize the need to retrieve specific pieces of information (snippets, contextual summaries, relevant book sections) rather than entire documents, reflecting user behavior in many search scenarios.
  • The creation of valuable test collections: A significant contribution of INEX is the ongoing development of diverse and challenging test collections that drive research in focused retrieval.

Key Results

  • Social Book Search Track: Eight teams submitted 33 runs for the SBS task, and two teams submitted 12 runs for the Prove It task. The best performing systems utilized all available book information and incorporated topic statement elements. No groups used user profile information.
  • Linked Data Track: Four ad-hoc search runs were submitted by three participants, and two valid Jeopardy! runs by one participant. The best scoring submission for the ad hoc task was from Renmin University of China (RUC) with a MAiP of 0.3880. The best scoring submission for the Jeopardy task was from the Max Planck Institute for Informatics (MPI) with an MRR of 0.7671.
  • Tweet Contextualization Track: A total of 13 teams submitted runs. The best performing system utilized hashtag preprocessing and web links (though this was disallowed in official runs). The second best run, without web links, ranked first among official runs.
  • Snippet Retrieval Track: Preliminary results indicated that the best scoring system achieved a GM score of 0.5352. The paper notes that poor snippets are causing users to miss relevant results, highlighting the need for further work in this area.

Limitations and Future Work

  • Low submission numbers: For some tracks, particularly the Linked Data track, the low number of submissions made it difficult to draw broad conclusions about system performance.
  • Need for further research in user modeling: The Social Book Search track noted that no groups utilized user profile information, suggesting this as a promising area for future work.
  • Improving snippet quality: The Snippet Retrieval track results indicate that significant work is still needed to ensure snippets effectively help users determine document relevance.
  • Exploring new tasks: The Social Book Search track outlook suggests exploring interactive sessions and extending tasks to require more detailed information about returned books. The Prove It task is not continuing, but new tasks around entity recognition are being considered.
  • Enhancing test collections: Future work for the Tweet Contextualization track aims to maintain stability in the use case and topic selection but explore more diverse tweet types to better measure the impact of hashtag processing. The Snippet Retrieval track also anticipates further document assessment to finalize results.

Summary updated

Abstract

INEX investigates focused retrieval from structured docu-ments by providing large test collections of structured documents, uni-form evaluation measures, and a forum for organizations to compare their results. This paper reports on the INEX 2014 evaluation cam-paign, which consisted of three tracks: The Interactive Social Book Search Track investigated user information seeking behavior when interacting with various sources of information, for realistic task scenarios, and how the user interface impacts search and the search experience. The Social Book Search Track investigated the relative value of authoritative meta-data and user-generated content for search and recommendation using a test collection with data from Amazon and LibraryThing, including user profiles and personal catalogues. The Tweet Contextualization Track in-vestigated tweet contextualization, helping a user to understand a tweet by providing him with a short background summary generated from rel-evant Wikipedia passages aggregated into a coherent summary. INEX 2014 was an exciting year for INEX in which we for the third time ran our workshop as part of the CLEF labs. This paper gives an overview of all the INEX 2014 tracks, their aims and task, the built test-collections, the participants, and gives an initial analysis of the results. 1
BibTeX
@inproceedings{bellot2013overview,
  author = {Patrice Bellot and Antoine Doucet and Shlomo Geva and Sairam Gurajada and Jaap Kamps and Gabriella Kazai and Marijn Koolen and Arunav Mishra and Véronique Moriceau and Josiane Mothe and Michael Preminger and Eric SanJuan and Ralf Schenkel and Xavier Tannier and Martin Theobald and Matthew Trappett and Qiuyue Wang},
  title = {{Overview of INEX 2013}},
  booktitle = {CLEF},
  year = {2013},
  doi = {10.1007/978-3-642-40802-1\_27},
}
Figure