Creating a Test Collection for Citation-based IR Experiments

Anna Ritchie, Simone Teufel, Stephen RobertsonNAACL 2006

TL;DR

This paper introduces a novel test collection for evaluating information retrieval (IR) systems that leverage citation links, a crucial aspect of scientific literature. By adapting the Cranfield methodology to the ACL Anthology, they've created a valuable resource for exploring citation-based IR, addressing a gap in existing benchmarks.

What Problem Does This Paper Address?

Traditional information retrieval (IR) systems primarily rely on the textual content of documents. However, in scientific literature, citations act as powerful links, connecting related research and providing valuable contextual information. Existing IR test collections, like those used in TREC, often focus on newspaper articles and lack the rich citation structure inherent in scientific papers. This makes it difficult to develop and evaluate IR systems that can effectively utilize citation information to improve retrieval. The authors aim to bridge this gap by creating a test collection specifically designed for citation-based IR experiments, allowing researchers to explore how citation links can enhance search.

Methods

The authors adapted the Cranfield 2 methodology for building test collections. Instead of manually creating queries and relevance judgments, they leveraged the existing structure of scientific papers and the authors' knowledge.

  • Document Collection: The ACL Anthology, a digital archive of computational linguistics research papers, was chosen as the source for the document collection. This was selected for its manageable size, relatively homogeneous field, and high proportion of internal references (citations within the collection).
  • Query and Relevance Judgement Elicitation: Authors of papers accepted to the ACL-2005 and HLT-EMNLP-2005 conferences were invited to provide their research questions and judge the relevance of papers cited in their own work. This approach treats each paper's underlying research question as a potential search query and uses the paper's reference list as a starting point for identifying relevant documents.
  • Relevance Scale: A 4-point relevance scale was used, which the authors felt was clearer than the 5-point scale used in Cranfield.
  • Deviation from Cranfield: Unlike the original Cranfield method, authors were not asked to list additional relevant papers not cited in their work. This was to maintain consistency and avoid subjective biases.
  • Document Processing: A pipeline was developed to convert PDF documents from the ACL Anthology into a usable textual format. This involved using OCR (OmniPage Pro) to extract character information, followed by pre-processing and conversion into a logical XML structure that annotates reference lists and citations.

Datasets

  • ACL Anthology: This served as the primary document collection, comprising approximately 7000 papers from the field of computational linguistics.
  • Newly Introduced Benchmark: The paper introduces a new test collection derived from the ACL Anthology, consisting of 170 queries with relevance judgments. This collection is specifically tailored for citation-based IR experiments.

Key Insights and Ideas

  • Citation as Anchor Text Analogy: The paper draws a strong parallel between citation links in scientific papers and hyperlinks on the web, suggesting that citation context can function similarly to anchor text in improving retrieval.
  • Author Expertise for Relevance Judgments: Leveraging the authors of scientific papers as judges for relevance is a practical and effective way to gather high-quality judgments, especially for specialized domains.
  • The "In-Factor" of a Collection: The concept of an "in-factor" (proportion of internal references) is crucial for building a self-contained test collection, which is beneficial for citation-based IR experiments. The ACL Anthology exhibits a relatively high in-factor.
  • Challenges of Low Relevance Counts: The initial results highlight a potential issue: a low number of relevant documents per query can negatively impact retrieval performance metrics. This underscores the importance of the planned second stage of relevance judgment collection.
  • Intrinsic Difficulty of Test Collections: The paper acknowledges that different test collections have varying intrinsic difficulties, making direct comparisons of absolute performance metrics across different collections (like TREC) potentially misleading. The focus should be on relative performance of retrieval variants.

Key Results

  • Test Collection Size: The initial collection comprises 170 unique queries with at least one relevant Anthology reference, averaging 3.8 relevant Anthology references per query.
  • Initial Retrieval Performance: Using the Indri IR system, the Mean Average Precision (MAP) was 0.1014, and R-precision was 0.0965 for the 151 queries with relevant references. Average precision at 5 documents was 0.0728.
  • Effect of Thresholding: Precision at 5 documents increased significantly when queries with a higher number of judged relevant documents were considered, suggesting that the low number of relevant documents is indeed impacting results.
  • Impact of Query Selection: Focusing on "main queries" (as opposed to all queries) led to an increase in MAP to 0.1165.

Limitations and Future Work

  • Incomplete Relevance Information: The authors acknowledge that their test collection is likely to suffer from incomplete relevance information, even after the planned second stage of data collection.
  • Second Stage of Relevance Judgments: A crucial next step is to re-approach contributing authors to judge the relevance of documents retrieved by standard IR models, and ideally, by citation-based methods.
  • Exploring bpref Measure: The paper suggests investigating the bpref measure, which is more stable with incomplete relevance judgments, as a potential solution to the incompleteness problem.
  • Adding External Documents: While considered, adding external documents to the collection is deemed less preferable as it would compromise the "identity" and "coverage of a field" properties of the ACL Anthology.
  • Future Experimentation: The authors intend to use this test collection for experiments involving citation information, aiming to improve term-based IR on scientific articles.

Summary updated

BibTeX
@inproceedings{ritchie2006creating,
  author = {Anna Ritchie and Simone Teufel and Stephen Robertson},
  title = {{Creating a Test Collection for Citation-based IR Experiments}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure