Report of the INEX 2003 metrics working group

Gabriella KazaiDuEPublico 2014

Abstract

This paper summarises the discussions of the metrics working group at the INEX 2003 Workshop, Dagstuhl, Dec 15-17 2003. Members of the group were Djoerd Hiemstra (U. of Twente), Jaap Kamps (ILLC, U. of Amsterdam), Gabriella Kazai (Queen Mary U. of London), Yosi Mass (IBM Haifa), Vojkan Mihajlovic (U. of Twente), Paul Ogilvie (Carnegie Mellon U.), Jovan Pehcevski (RMIT U.), Arjen de Vries (CWI) and Huyen-Trang Vu (LIP 6).

Introduction

The evaluation of XML retrieval systems presents unique challenges compared to traditional Information Retrieval (IR). While standard web search treats documents as atomic units, XML retrieval aims to identify specific document components—such as paragraphs or sections—that directly answer a query. This granularity introduces two primary difficulties: the "overlap problem" and the complexity of two-dimensional relevance.

The overlap problem occurs when systems are rewarded for returning multiple nested components (e.g., a paragraph and its parent section) that contain the same information. Traditional metrics often fail by treating these as independent hits. Furthermore, relevance in XML is defined by both Exhaustivity (the extent to which an element covers the topic) and Specificity (the extent to which an element is focused solely on the topic).

Proposed Evaluation Metrics

To address these challenges, the working group reviewed several novel metrics designed to model real-world user behavior and structural constraints:

  • inex-2003 Metric: An evolution of the previous year's metric that incorporates component size. It rewards systems only for "new" information, effectively penalizing the retrieval of overlapping nested structures.
  • Expected Ratio of Relevant (ERR): A probabilistic metric that models a user browsing through "doxels" (document elements). It calculates the expected ratio of relevant elements encountered relative to the total available.
  • Tolerance to Irrelevance (T2I): A user-centric metric based on the concept of an "entry point." It assumes a user reads from a suggested starting point until a threshold of irrelevant content is reached.

Key Insights and Results

A central realization of the group was that retrieval should be viewed as a trade-off between information gain and user effort. This led to a preference for metrics that penalize overlap by crediting only incremental information. The group analyzed various strategies for selecting the most appropriate elements from a document hierarchy to maximize precision and recall.

Precision-recall curves comparing different element selection strategies.

The group also explored quantization functions to map two-dimensional relevance scores into a single ranking value. While "strict" quantization (rewarding only highly exhaustive and specific elements) simplifies the overlap issue, "generalized" functions better capture the nuances of content-oriented search. The working group reached a consensus on a roadmap for future evaluations, mapping specific metrics to the various XML retrieval tasks.

Table 1: Recommended metrics for INEX tasks.

Limitations and Future Work

Despite progress, several challenges remain. Metrics like ERR involve complex parameters that are difficult to estimate. Additionally, the high cognitive load placed on human assessors to provide graded relevance scores (0–3 scales) can lead to inconsistencies in ground-truth data. A recurring issue is the "article baseline," where returning an entire article often outperforms more granular retrieval. Future research must refine metrics to better reward the identification of the "perfect" small element without being overly sensitive to minor ranking fluctuations.

Summary updated

Abstract

This paper summarises the discussions of the metrics working group at the INEX 2003 Workshop, Dagstuhl, Dec 15-17 2003. Members of the group were Djoerd Hiemstra (U. of Twente), Jaap Kamps (ILLC, U. of Amsterdam), Gabriella Kazai (Queen Mary U. of London), Yosi Mass (IBM Haifa), Vojkan Mihajlovic (U. of Twente), Paul Ogilvie (Carnegie Mellon U.), Jovan Pehcevski (RMIT U.), Arjen de Vries (CWI) and Huyen-Trang Vu (LIP 6)
BibTeX
@inproceedings{kazai2014report,
  author = {Gabriella Kazai},
  title = {{Report of the INEX 2003 metrics working group}},
  booktitle = {DuEPublico},
  year = {2014},
}
Figure