CSFCube: A Test Collection for Faceted Query by Example in Computer Science
Abstract
Query by Example (QBE) is a critical information retrieval task where a user selects a document to find similar ones. However, scientific papers are multi-faceted, covering distinct objectives, methods, and results. We introduce CSFCube, an expert-annotated test collection designed for faceted QBE. By allowing users to specify a finer-grained aspect (facet) alongside the query document, CSFCube enables more precise retrieval. Our evaluation of state-of-the-art models reveals a significant performance gap, particularly in retrieving papers with similar methodologies.
Introduction: The Multi-Faceted Nature of Research
Traditional keyword-based search often falls short during exploratory research when users cannot easily articulate specific technical terms. Query by Example (QBE) addresses this by using a document as the query. Yet, a researcher might be interested in a paper's methodology but not its application domain, or vice versa. Standard QBE models often conflate these aspects, returning papers that are broadly "topically" similar but technically mismatched.
Existing datasets for scientific retrieval often rely on noisy proxies like citation networks or are too small for robust evaluation. To bridge this gap, we formalize the task of faceted QBE, where retrieval is conditioned on specific rhetorical elements: Background/Objective, Method, and Result.
The CSFCube Dataset
To build a high-quality benchmark, we selected 50 query abstracts from top-tier NLP and ML venues. We employed a "pooling" strategy, aggregating results from six diverse retrieval methods (including TF-IDF, SPECTER, and citation-based models) to create a candidate pool of over 6,000 pairs from the S2ORC corpus.
Crucially, four graduate-level computer science experts manually rated these candidates on a graded scale (0-3) for each facet. This ensures the dataset reflects deep technical understanding rather than simple keyword overlap.
Key Insights and Model Performance
Our analysis of the expert annotations and model evaluations yielded several critical findings:
- The "Method" Gap: Current state-of-the-art models like SPECTER-ID perform reasonably well at finding papers with similar backgrounds (NDCG@20 ~69%) but struggle significantly with the Method facet (NDCG@20 ~42%). This suggests that while models capture general topics, they fail to grasp mechanistic or procedural similarity.
- Facet Independence: The low correlation between relevance scores across different facets confirms that these rhetorical elements represent distinct dimensions of similarity. A paper relevant in "Background" is not necessarily relevant in "Method."
- Citations as Noisy Proxies: Only 12-21% of cited papers were rated as highly relevant for a specific facet, highlighting that citation networks are insufficient for capturing fine-grained semantic similarity.
- Negation and Logic: Error analysis showed that models often fail to distinguish between opposing intents (e.g., "improving X" vs. "shortcomings in X") when they share the same technical keywords.
Conclusion and Future Work
CSFCube highlights the limitations of current scientific document embedding models. While whole-abstract models generally outperform sentence-level ones, there is a clear need for architectures that can be "conditioned" on specific facets. Future work should focus on scaling expert annotations and incorporating full-text features to better capture complex methodological similarities.
The dataset and guidelines are publicly available at https://github.com/iesl/CSFCube.
Summary updated