CSFCube - A Test Collection of Computer Science Research Articles for Faceted Query by Example

Sheshera Mysore, Tim O'Gorman, Andrew McCallum, Hamed Zamani

CSFCube: A Test Collection for Faceted Query by Example in Computer Science

Abstract

Query by Example (QBE) is a critical information retrieval task where a user selects a document to find similar ones. However, scientific papers are multi-faceted, covering distinct objectives, methods, and results. We introduce CSFCube, an expert-annotated test collection designed for faceted QBE. By allowing users to specify a finer-grained aspect (facet) alongside the query document, CSFCube enables more precise retrieval. Our evaluation of state-of-the-art models reveals a significant performance gap, particularly in retrieving papers with similar methodologies.

Introduction: The Multi-Faceted Nature of Research

Traditional keyword-based search often falls short during exploratory research when users cannot easily articulate specific technical terms. Query by Example (QBE) addresses this by using a document as the query. Yet, a researcher might be interested in a paper's methodology but not its application domain, or vice versa. Standard QBE models often conflate these aspects, returning papers that are broadly "topically" similar but technically mismatched.

Existing datasets for scientific retrieval often rely on noisy proxies like citation networks or are too small for robust evaluation. To bridge this gap, we formalize the task of faceted QBE, where retrieval is conditioned on specific rhetorical elements: Background/Objective, Method, and Result.

The CSFCube Dataset

To build a high-quality benchmark, we selected 50 query abstracts from top-tier NLP and ML venues. We employed a "pooling" strategy, aggregating results from six diverse retrieval methods (including TF-IDF, SPECTER, and citation-based models) to create a candidate pool of over 6,000 pairs from the S2ORC corpus.

Faceted Query by Example: A query paper is compared against candidates based on specific facets like background and method.

Crucially, four graduate-level computer science experts manually rated these candidates on a graded scale (0-3) for each facet. This ensures the dataset reflects deep technical understanding rather than simple keyword overlap.

Key Insights and Model Performance

Our analysis of the expert annotations and model evaluations yielded several critical findings:

  • The "Method" Gap: Current state-of-the-art models like SPECTER-ID perform reasonably well at finding papers with similar backgrounds (NDCG@20 ~69%) but struggle significantly with the Method facet (NDCG@20 ~42%). This suggests that while models capture general topics, they fail to grasp mechanistic or procedural similarity.
  • Facet Independence: The low correlation between relevance scores across different facets confirms that these rhetorical elements represent distinct dimensions of similarity. A paper relevant in "Background" is not necessarily relevant in "Method."
  • Citations as Noisy Proxies: Only 12-21% of cited papers were rated as highly relevant for a specific facet, highlighting that citation networks are insufficient for capturing fine-grained semantic similarity.
  • Negation and Logic: Error analysis showed that models often fail to distinguish between opposing intents (e.g., "improving X" vs. "shortcomings in X") when they share the same technical keywords.
Heatmaps showing the distribution of expert relevance scores across different facet pairs (Background, Method, Result).

Conclusion and Future Work

CSFCube highlights the limitations of current scientific document embedding models. While whole-abstract models generally outperform sentence-level ones, there is a clear need for architectures that can be "conditioned" on specific facets. Future work should focus on scaling expert annotations and incorporating full-text features to better capture complex methodological similarities.

The dataset and guidelines are publicly available at https://github.com/iesl/CSFCube.

Summary updated

Abstract

Query by Example is a well-known information retrieval task in which a document is chosen by the user as the search query and the goal is to retrieve relevant documents from a large collection. However, a document often covers multiple aspects of a topic. To address this scenario we introduce the task of faceted Query by Example in which users can also specify a finer grained aspect in addition to the input query document. We focus on the application of this task in scientific literature search. We envision models which are able to retrieve scientific papers analogous to a query scientific paper along specifically chosen rhetorical structure elements as one solution to this problem. In this work, the rhetorical structure elements, which we refer to as facets, indicate objectives, methods, or results of a scientific paper. We introduce and describe an expert annotated test collection to evaluate models trained to perform this task. Our test collection consists of a diverse set of 50 query documents in English, drawn from computational linguistics and machine learning venues. We carefully follow the annotation guideline used by TREC for depth-k pooling (k = 100 or 250) and the resulting data collection consists of graded relevance scores with high annotation agreement. State of the art models evaluated on our dataset show a significant gap to be closed in further work. Our dataset may be accessed here: https://github.com/iesl/CSFCube
BibTeX
@inproceedings{mysore2021csfcube,
  author = {Sheshera Mysore and Tim O'Gorman and Andrew McCallum and Hamed Zamani},
  title = {{CSFCube - A Test Collection of Computer Science Research Articles for Faceted Query by Example}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=BZ_gcqpvQGB},
}
Figure