Abstract
Scientific literature exists within a complex ecosystem of textual content and citation networks. While traditional Single Document Representation Learning (SDRL) focuses on isolated papers, Multi-Document Representation Learning (MDRL) remains a challenge. We present SMRC², a novel framework that jointly models semantic content and topological relatedness. By employing a Wasserstein Distance constraint to align language models with entity-level graph neural networks and a masking strategy for citation recovery, SMRC² achieves state-of-the-art results across document classification, citation prediction, and multi-document summarization tasks.
Introduction
In scientific research, documents are interconnected nodes in a vast network of citations and shared entities. Most current NLP tools are designed for Single Document Representation Learning (SDRL), focusing either on the textual content or the document's position within a citation graph. However, text can be noisy, and citation graphs often lack the semantic nuance of the research itself. Bridging this gap requires a unified approach that captures both the "what" (content) and the "how" (relationships) of scientific literature at a multi-document level.
Methodology: The SMRC² Framework
The SMRC² (Scientific Multi-literature Representation) framework is composed of two synergistic modules designed to learn comprehensive document embeddings.
1. Semantic Information Learning
This module focuses on the internal content of papers. It utilizes a Language Model (LM) to encode abstracts and a Graph Neural Network (GNN) to capture "entity-level" information—key terms and their relationships within the document. To ensure these two distinct views inform each other, the authors introduce a Wasserstein Distance (WD) constraint. Unlike standard KL-Divergence, WD provides a more stable measure of discrepancy between distributions, facilitating interactive learning between the LM and GNN.
2. Relatedness Information Learning
This module models the broader scientific ecosystem by treating groups of papers as a citation graph. Using a masking strategy inspired by graph auto-encoders, the model randomly hides citation links. A GNN-based decoder is then tasked with recovering these links based on learned semantic features, forcing the model to internalize the underlying logic of document relatedness.
Experimental Setup and Datasets
The framework was evaluated on four diverse datasets:
- Ogbn-Arxiv: A benchmark for paper classification.
- Ogbl-Citation2: A benchmark for citation prediction.
- Multi-XScience: A large-scale dataset for multi-document summarization.
- Bio-Sci: A new dataset introduced in this work, comprising 32,330 publications and 222,652 citation relations across 11 biomedical domains.
Results and Discussion
SMRC² consistently outperformed state-of-the-art baselines, including specialized models like Bio-Bert, Sci-Bert, and SPECTER.
Key Performance Metrics:
- Document Classification: SMRC² achieved a mean accuracy of 88.41% on the Bio-Sci dataset, significantly surpassing the best graph-only baseline (EnGCN at 85.27%).
- Citation Prediction: The model demonstrated superior performance on both Ogbn and Bio-Sci benchmarks compared to standard GCN and GraphSage architectures.
- Multi-Document Summarization: On the Multi-XScience dataset, SMRC² reached a ROUGE-1 score of 31.47, outperforming specialized summarization models like KGSum and REFLECT.
The success of SMRC² highlights the importance of the Wasserstein constraint in balancing semantic and entity-level data, as well as the effectiveness of capturing cross-correlations across GNN layers for topological reconstruction.
Conclusion and Future Work
SMRC² represents a significant step forward in MDRL by effectively integrating content and topology. While the model currently requires pre-existing citation links and carries a high computational cost due to its multi-module architecture, it sets a new standard for scientific document representation. Future research will focus on improving scalability and enhancing the interpretability of the high-dimensional concatenated embeddings.
Summary updated