TL;DR
CE-GOCD improves Large Language Model (LLM) scientific question answering by organizing academic knowledge graphs into structured subgraphs anchored around paper titles. By optimizing these subgraphs through semantic pruning and implicit relation completion, and then clustering them into thematic communities, it enables LLMs to synthesize comprehensive answers across multiple research papers.
What Problem Does This Paper Address?
When researchers ask complex questions about scientific literature, standard Retrieval-Augmented Generation (RAG) methods often fall short. Typical RAG pipelines split text into isolated chunks or focus on standalone concepts, completely missing the deeper, interconnected relationships between papers (e.g., how a specific methodology from one paper resolves a problem introduced by another).
While structured Knowledge Graphs (KGs) are excellent at capturing explicit conceptual relations, they struggle to map the implicit connections hidden within scientific literature. This lack of multi-source information fusion leads to LLM responses that are either incomplete, superficial, or prone to hallucination. The authors tackle this gap by proposing a method that explicitly models and leverages the semantic substructures connecting academic publications.
Methods
The authors introduce CE-GOCD (Central Entity-Guided Graph Optimization for Community Detection), a framework that operates in three main stages:
- Central Entity-Guided Subgraph Retrieval: Using an LLM, the system extracts contextual keywords and target entity types from a user's query. It retrieves relevant entities from an academic knowledge graph and constructs a localized subgraph. Crucially, it treats paper titles as "central entities" that anchor all related concepts and paper elements together.
- Subgraph Optimization: The retrieved subgraph is refined using two techniques:
- Pruning: Edges with weak relevance to the query are removed based on a combination of semantic similarity and relation types, using a self-adaptive threshold.
- Completion: To uncover hidden, implicit relations (like similar tasks or methodologies across different papers), entity embeddings are projected into a 1D semantic space. Proximity analysis and an LLM are then used to infer and add missing correlations.
- Community Detection: The optimized graph is partitioned into cohesive clusters using the modularity-based Louvain algorithm. For each community, a central paper title node with the highest weighted degree is identified. These structured communities are translated into natural language descriptions, allowing the LLM to generate targeted, community-level answers before synthesizing them into a final response.
Datasets
The framework was evaluated using NLP-AKG, a large-scale academic knowledge graph constructed from 61,826 ACL Anthology papers containing over 620,000 entities and 2.2 million relations.
For the question-answering evaluation, the authors utilized three datasets:
- QASPER: A standard benchmark for long-document scientific question answering.
- PeerQA: A dataset featuring questions sourced from academic peer reviews, with answers provided by the original paper authors.
- NLP-MQA: A newly introduced dataset annotated by the authors, consisting of 200 multi-paper question-answer pairs within the NLP domain, created specifically to address the shortage of open-source multi-paper QA benchmarks.
Key Insights and Ideas
- Titles as Semantic Anchors: Treating paper titles as central hubs within the knowledge graph effectively bridges the gap between high-level document metadata and granular, isolated concepts.
- Latent Link Discovery: Subgraph completion via 1D embedding projections allows the system to discover latent connections—such as two distinct papers utilizing similar algorithmic tricks—that aren't explicitly linked in the base knowledge graph.
- Mitigating Information Overload: Instead of feeding a massive, messy graph directly to an LLM, clustering the graph into distinct thematic communities (typically capped at 3) allows the LLM to reason systematically across different research angles without getting overwhelmed by context length.
Key Results
CE-GOCD consistently outperformed state-of-the-art retrieval augmentation baselines (including KAG, MindMap, and PathRAG) across three different LLMs: GPT-4, DeepSeek-V3, and Qwen-plus.
- NLP-MQA: Achieved the highest F1-score across all models, reaching 79.01% with GPT-4 (an 8.95% improvement over the LLM-only baseline) and 80.14% with DeepSeek-V3.
- QASPER: Demonstrated an F1 improvement of +9.76% using DeepSeek-V3 and +7.70% with Qwen-plus.
- Domain Generalizability: When tested on a medical dataset (ChatDoctor5K), the method remained highly competitive with specialized baselines, proving its potential to generalize outside of the NLP domain.
Limitations and Future Work
- Computational Overhead: The multi-step graph optimization and community detection pipeline results in an average response time of 129 seconds per query. While it balances depth of analysis well, this latency may be too high for real-time interactive applications.
- Domain Structural Sensitivity: Although the model generalized well to the medical domain, it did not completely outperform all specialized medical baselines. The authors note that differences in conceptual structures between highly structured scientific literature and medical clinical data present unique challenges that require further refinement.
Summary updated