Abstract
Because most of the scientific literature data is unmarked, it makes semantic representation learning based on unsupervised graph become crucial. At the same time, in order to enrich the features of scientific literature, a learning method of semantic representation of scientific literature based on adaptive features and graph neural network is proposed. By introducing the adaptive feature method, the features of scientific literature are considered globally and locally. The graph attention mechanism is used to sum the features of scientific literature with citation relationship, and give each scientific literature different feature weights, so as to better express the correlation between the features of different scientific literature. In addition, an unsupervised graph neural network semantic representation learning method is proposed. By comparing the mutual information between the positive and negative local semantic representation of scientific literature and the global graph semantic representation in the potential space, the graph neural network can capture the local and global information, thus improving the learning ability of the semantic representation of scientific literature. The experimental results show that the proposed learning method is competitive on the basis of scientific literature classification and has achieved good results.
Introduction
The exponential growth of scientific literature has made manual categorization and search increasingly difficult. While deep learning has significantly advanced natural language processing, scientific documents present unique challenges: a lack of labeled data, complex relational contexts within citation graphs, and feature sparsity in traditional models. Supervised Graph Neural Networks (GNNs) often become too specialized for specific tasks, losing the generality required for broader applications like search or recommendation.
To address these issues, we propose AFPGNN (Adaptive Feature and Graph Neural Network), an unsupervised framework designed to learn high-quality semantic representations. By combining adaptive feature pooling with a graph attention mechanism and maximizing mutual information between local and global graph structures, AFPGNN effectively captures the nuances of scientific text and citation relationships without requiring labeled data.
Methodology
The AFPGNN architecture consists of three innovative pillars designed to capture both the content and the context of scientific papers.
1. Adaptive Feature Module
Instead of relying on a single raw vector, the model enriches document features using a weighted combination of original data, maximum pooling (to capture local, salient features), and average pooling (to capture global context). This module is parameter-free, which significantly reduces computational overhead while addressing feature sparsity.
2. Graph Attention Encoder
Not all citations are equally relevant to a paper's core topic. AFPGNN utilizes a multi-head attention mechanism to assign dynamic weights to neighboring nodes in the citation graph. This allows the model to "focus" on the most semantically relevant related works when defining a paper's identity.
3. Maximum Mutual Information (MMI) Learning
To facilitate unsupervised training, the model employs a contrastive strategy. It compares a "local" node representation with a "global" summary of the entire graph. By maximizing the mutual information between these two, the GNN learns to capture structural patterns that are consistent across the whole network.
Experimental Evaluation
The model was evaluated on two gold-standard citation network benchmarks: Cora (2,708 papers, 5,429 links) and Citeseer (3,327 papers, 4,732 links). These datasets were used to test how well the learned unsupervised representations perform on downstream document classification tasks.
Key Results
AFPGNN demonstrated superior performance across multiple metrics, including Accuracy, Macro-F1, and Recall. On the Cora dataset, AFPGNN achieved an accuracy of 83.4%, outperforming popular models like GraphSAGE (81.9%) and GCN (81.5%).
Ablation and Stability
Ablation tests confirmed that the combination of all three modules is essential for peak performance. For instance, removing the attention mechanism or the mutual information objective led to significant drops in accuracy. Furthermore, hyperparameter analysis revealed that using 6 attention heads and a high dropout rate (0.8) provided the most stable and accurate training process.
Conclusion
AFPGNN provides a robust, unsupervised framework for learning scientific literature representations. By balancing local and global features through adaptive pooling and attention-weighted graph aggregation, it captures the complex semantics inherent in research networks. Future work will explore the model's scalability to larger graphs and its performance in dynamic citation environments.
Summary updated