TL;DR
The INEX 2010 workshop brought together researchers to evaluate focused retrieval techniques on structured documents, particularly XML. This report details the various tracks, including Ad Hoc, Book, Data Centric, Interactive, QA, Link the Wiki, Relevance Feedback, Web Service Discovery, and XML Mining, highlighting their aims, methodologies, and key findings, and emphasizing the creation of valuable test collections for future research.
What Problem Does This Paper Address?
Traditional information retrieval (IR) often treats documents as "bags of words," overlooking valuable structural and semantic information. As the web and professional search increasingly rely on structured data (like XML, metadata, and entities), there's a growing need for "focused retrieval" – systems that can pinpoint the most relevant parts of documents rather than just entire documents. INEX has been a pioneer in this area since 2002, providing a platform for researchers to develop and evaluate such systems using standardized measures and test collections. The INEX 2010 workshop aimed to further this by exploring focused retrieval across a diverse range of tasks and data types, including books, databases, and web services, and by investigating user interaction with these systems.
Methods
INEX 2010 featured several tracks, each with its own specific methodologies:
- Ad Hoc Track: Focused on XML-IR and passage retrieval, using Wikipedia as a corpus. It explored resource-restricted retrieval (e.g., for mobile devices) and efficiency-effectiveness trade-offs.
- Book Track: Evaluated approaches for searching and navigating digitized books, with tasks like building reading lists, verifying factual claims, and extracting structure.
- Data Centric Track: Investigated retrieval over highly structured database data (IMDb), comparing whole-document retrieval with focused retrieval.
- Interactive Track (iTrack): Studied user interaction with XML-based IR systems, focusing on how users exploit document parts and full documents.
- Question Answering (QA) Track: Aimed to compare QA, XML, and passage retrieval systems for answering factual and complex questions, often requiring multi-document aggregation.
- Link the Wiki Track: Focused on link discovery within encyclopedic content (Te Ara), moving beyond Wikipedia to explore new challenges.
- Relevance Feedback Track: Simulated user interaction with IR systems, where users provide feedback on retrieved passages to improve subsequent results.
- Web Service Discovery Track: Addressed the discovery of web services based on their WSDL descriptions.
- XML Mining Track: Explored data mining techniques (clustering and classification) for structured XML documents.
A common thread across many tracks was the use of XML-enriched collections and the development of evaluation measures that go beyond traditional precision and recall to account for factors like result length, reading effort, and the specificity of retrieved information. Crowdsourcing (e.g., Amazon's Mechanical Turk) was also employed for topic creation and relevance assessment in some tracks.
Datasets
- Wikipedia: Used extensively for the Ad Hoc and QA tracks, with versions updated over several years and enriched with semantic markup (YAGO).
- Digitized Books: A collection of over 50,000 out-of-copyright books was used for the Book Track.
- IMDb: A large collection of XML files representing movies and persons was used for the Data Centric Track.
- Amazon and LibraryThing: Data from these sources was used for the Interactive Track.
- Te Ara Encyclopedia: A collection of 36,715 documents was used for the Link the Wiki Track.
- WSDL Documents: A collection of 1,738 WSDL documents was used for the Web Service Discovery Track.
Several tracks also introduced new test collections or significantly modified existing ones, often leveraging crowdsourcing for data creation and assessment.
Key Insights and Ideas
- The Importance of Structure: The consistent success and focus of INEX highlight that structured information (XML, metadata, semantic links) is crucial for effective focused retrieval, moving beyond simple text matching.
- Resource-Restricted Retrieval is Key: The Ad Hoc track's emphasis on mobile devices and summary-like results underscores the need for systems that can deliver concise, highly relevant information efficiently.
- Crowdsourcing for Data Creation: The successful use of platforms like Amazon Mechanical Turk for generating topics and relevance judgments demonstrates a viable path for creating large-scale, high-quality datasets with reduced reliance on expert assessors.
- Beyond Document-Level Retrieval: Many tracks explored novel evaluation metrics and tasks that acknowledge the granularity of information needs, such as passage retrieval, snippet generation, and identifying specific book sections.
- Challenges in Link Discovery: Moving from Wikipedia to a less structured encyclopedia like Te Ara presented new challenges for link discovery, indicating that existing methods may need adaptation for different types of linked data.
Key Results
- Ad Hoc Track: Explored effectiveness under resource constraints, with results showing that while pinpointing exact text is improving, locating the right articles remains a prerequisite. Efficiency trade-offs were also studied, with some systems achieving very low response times.
- Book Track: The Best Books to Reference (BB) task saw top runs achieving NDCG@10 scores around 0.65, with approaches combining page-level retrieval and book-level aggregation performing well. The Prove It (PI) task also saw competitive results, with individual page indexing and pseudo relevance feedback showing promise.
- Data Centric Track: The research question of whether whole-document retrieval outperforms focused retrieval on structured data was investigated, returning to INEX's roots.
- Interactive Track: Data from 147 sessions involving 49 test subjects was collected, with analysis focusing on searcher choices and the influence of task types.
- Question Answering Track: Evaluation of long answers used Kullback-Leibler divergence, with systems performing best when questions referred to specific Wikipedia pages.
- Link the Wiki Track: The best run achieved a mean average precision of 0.3469 in identifying relevant anchors and targets within the Te Ara encyclopedia.
- Relevance Feedback Track: Results showed that incremental feedback significantly improved ranking performance compared to initial rankings.
- Web Service Discovery Track: The best performing run achieved a mean average precision of 0.3469.
- XML Mining Track: Clustering and classification tasks were evaluated, with some submissions showing improved performance as the number of clusters increased.
Limitations and Future Work
- Ad Hoc Track: The paper notes that the current Wikipedia collection might not be a sufficient challenge for efficiency due to low response times. Future work will likely involve new tasks like Social Search and Faceted Search.
- Book Track: The BB task is being replaced by a new Social Search for Best Books (SSBB) task in 2011, which will incorporate user-generated metadata. The PI task will continue, asking systems to differentiate positive and negative evidence.
- Data Centric Track: Future work aims to improve query and structure integration for better performance.
- Interactive Track: Data is still being analyzed, with a focus on searcher source selection and task type influence.
- Question Answering Track: Future work will involve mixing short and long answer questions, annotating Wikipedia dumps for QA, and having systems infer answer type and context.
- Link the Wiki Track: Future work will explore cross-lingual link discovery.
- Web Service Discovery Track: Future work aims to allow topics to include workflow descriptions for finding web services that fit into larger processes.
- XML Mining Track: Future work will likely focus on further exploring clustering and classification techniques for structured data.
Overall, INEX 2010 successfully advanced the field of focused retrieval by creating valuable test collections and fostering research across a wide array of structured data and information needs.
Summary updated