TL;DR
This paper introduces a new task: aligning scientific papers to specific topics within a research proposal. They've created a dataset and a novel three-stage approach using smaller language models to achieve this, showing comparable results to larger models while highlighting the task's inherent complexity.
What Problem Does This Paper Address?
When researchers write proposals, they need to connect their ideas to existing scientific literature. While tools can retrieve relevant papers based on a proposal's abstract, a deeper level of organization is often needed: mapping these papers to high-level thematic categories (topics) defined by the researcher. This fine-grained alignment helps in motivating research problems, identifying novelty, and building literature reviews. Existing methods for automatic topic categorization often fail to align with researcher-defined categories, leading to discrepancies. This paper tackles the problem of automatically aligning relevant scientific articles to researcher-provided topics, especially in the early stages of proposal writing where citation text might not be readily available.
Methods
The authors propose a novel three-stage approach:
- Data Augmentation: Since ground truth labels for text spans relevant to topics are scarce, they use citation text (available during training) as a link to retrieve relevant text spans from reference papers. They also generate pseudo-negative text spans to train the models.
- Training: They train two main components:
- Topic-Based Reference Text Span Retriever (TR): A language model that retrieves relevant text spans from a reference paper for a given topic.
- Reference Paper Classifier (TC): A language model that classifies whether a reference paper is relevant to a proposal topic, using the retrieved text spans.
- Inference: During inference, the TR component retrieves text spans from reference papers, and the TC component then classifies the paper's relevance to the proposal topic based on these spans. They use smaller, open-source language models (like T5 and RoBERTa) fine-tuned on their constructed dataset.
Datasets
The authors introduce a new dataset for this task, extending the UnArXiv dataset. They leverage ArXiv tags to focus on AI and Computer Vision domains. Their dataset consists of:
- Target proposals (simulated from UnArXiv papers).
- Topics (extracted from subsection headings in the 'Related Work' sections of these proposals).
- Reference papers (cited within these sections).
- Citation texts (linking topics to reference papers).
- Synthesized reference text spans (derived from citation texts).
They provide statistics on the dataset size, including the number of proposals, topics, reference papers, and topic-paper pairs across train, validation, and test splits.
Key Insights and Ideas
- Bridging the Gap: The core idea is to move beyond simple paper retrieval to a more nuanced topic-based alignment, crucial for structured scientific writing.
- Leveraging Citation Text: Effectively using citation text as a "link" to generate pseudo-labels for training is a key innovation, especially given the scarcity of direct annotations.
- Smaller Models, Big Impact: Demonstrating that smaller, fine-tuned open-source LMs can achieve performance comparable to larger, paid LLMs highlights the value of their dataset and approach.
- Task Complexity: The significant gap between their model's performance and human baselines underscores the inherent difficulty and subjectivity of aligning scientific content to specific research topics.
Key Results
- Their proposed pipeline, using smaller LMs trained on synthesized data, achieved results comparable to a vanilla paid LLM-based classifier.
- A notable gap of 23.57 F1 score exists between their approach and the human baseline, indicating room for improvement.
- Human baselines achieved high F1 scores, confirming the feasibility of the task.
- Their TR + LLM approach outperformed the vanilla LLM baseline, showing the benefit of using their retrieval model.
Limitations and Future Work
- Domain Specificity: The pipeline is fine-tuned on AI (majorly CV) papers. Its performance might degrade in cross-domain settings, though the dataset construction method is adaptable.
- Information Source: The approach primarily relies on citation text and section headings. The authors acknowledge that other sections might contain relevant information, but they lack explicit signals to exploit them.
- Topic Hierarchy: Capturing the hierarchy among topics (catalogue) was challenging due to PDF parsing limitations, treating each topic as standalone.
- Future Work: The authors suggest improving PDF parsing and semantic paragraph segmentation to capture topic hierarchies. They also propose augmenting the pipeline with an entailment task to better evaluate the relevance of retrieved text spans and address issues of subjectivity and insufficiency of information.
Summary updated