Content- and Topology-Aware Representation Learning for Scientific Multi-Literature

Kai Zhang, Kaisong Song, Yangyang Kang, Xiaozhong Liu

Abstract

Scientific literature exists within a complex ecosystem of textual content and citation networks. While traditional Single Document Representation Learning (SDRL) focuses on isolated papers, Multi-Document Representation Learning (MDRL) remains a challenge. We present SMRC², a novel framework that jointly models semantic content and topological relatedness. By employing a Wasserstein Distance constraint to align language models with entity-level graph neural networks and a masking strategy for citation recovery, SMRC² achieves state-of-the-art results across document classification, citation prediction, and multi-document summarization tasks.

Introduction

In scientific research, documents are interconnected nodes in a vast network of citations and shared entities. Most current NLP tools are designed for Single Document Representation Learning (SDRL), focusing either on the textual content or the document's position within a citation graph. However, text can be noisy, and citation graphs often lack the semantic nuance of the research itself. Bridging this gap requires a unified approach that captures both the "what" (content) and the "how" (relationships) of scientific literature at a multi-document level.

Methodology: The SMRC² Framework

The SMRC² (Scientific Multi-literature Representation) framework is composed of two synergistic modules designed to learn comprehensive document embeddings.

The SMRC² architecture, illustrating the Semantic Information Learning and Relatedness Information Learning modules.

1. Semantic Information Learning

This module focuses on the internal content of papers. It utilizes a Language Model (LM) to encode abstracts and a Graph Neural Network (GNN) to capture "entity-level" information—key terms and their relationships within the document. To ensure these two distinct views inform each other, the authors introduce a Wasserstein Distance (WD) constraint. Unlike standard KL-Divergence, WD provides a more stable measure of discrepancy between distributions, facilitating interactive learning between the LM and GNN.

2. Relatedness Information Learning

This module models the broader scientific ecosystem by treating groups of papers as a citation graph. Using a masking strategy inspired by graph auto-encoders, the model randomly hides citation links. A GNN-based decoder is then tasked with recovering these links based on learned semantic features, forcing the model to internalize the underlying logic of document relatedness.

Experimental Setup and Datasets

The framework was evaluated on four diverse datasets:

  • Ogbn-Arxiv: A benchmark for paper classification.
  • Ogbl-Citation2: A benchmark for citation prediction.
  • Multi-XScience: A large-scale dataset for multi-document summarization.
  • Bio-Sci: A new dataset introduced in this work, comprising 32,330 publications and 222,652 citation relations across 11 biomedical domains.

Results and Discussion

SMRC² consistently outperformed state-of-the-art baselines, including specialized models like Bio-Bert, Sci-Bert, and SPECTER.

Performance comparison of SMRC² against LM and Graph baselines for document classification.

Key Performance Metrics:

  • Document Classification: SMRC² achieved a mean accuracy of 88.41% on the Bio-Sci dataset, significantly surpassing the best graph-only baseline (EnGCN at 85.27%).
  • Citation Prediction: The model demonstrated superior performance on both Ogbn and Bio-Sci benchmarks compared to standard GCN and GraphSage architectures.
  • Multi-Document Summarization: On the Multi-XScience dataset, SMRC² reached a ROUGE-1 score of 31.47, outperforming specialized summarization models like KGSum and REFLECT.

The success of SMRC² highlights the importance of the Wasserstein constraint in balancing semantic and entity-level data, as well as the effectiveness of capturing cross-correlations across GNN layers for topological reconstruction.

Conclusion and Future Work

SMRC² represents a significant step forward in MDRL by effectively integrating content and topology. While the model currently requires pre-existing citation links and carries a high computational cost due to its multi-module architecture, it sets a new standard for scientific document representation. Future research will focus on improving scalability and enhancing the interpretability of the high-dimensional concatenated embeddings.

Summary updated

Abstract

Representation learning forms an essential building block in the development of natural language processing architectures. To date, mainstream approaches focus on learning textual information at the sentence- or document-level, unfortunately, overlooking the inter-document connections. This omission decreases the potency of downstream applications, particularly in multi-document settings. To address this issue, embeddings equipped with latent semantic and rich relatedness information are needed. In this paper, we propose SMRC2, which extends representation learning to the multi-document level. Our model jointly learns latent semantic information from content and rich relatedness information from topological networks. Unlike previous studies, our work takes multi-document as input and integrates both semantic and relatedness information using a shared space via language model and graph structure. Our extensive experiments confirm the superiority and effectiveness of our approach. To encourage further research in scientific multi-literature representation learning, we will release our code and a new dataset from the biomedical domain.
BibTeX
@inproceedings{zhang2023content,
  author = {Kai Zhang and Kaisong Song and Yangyang Kang and Xiaozhong Liu},
  title = {{Content- and Topology-Aware Representation Learning for Scientific Multi-Literature}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.465},
}
Figure