TL;DR
This paper introduces HetGCoT, a novel framework that bridges the gap between large language models (LLMs) and academic knowledge graphs for question answering. By transforming complex graph structures into natural language reasoning chains and employing adaptive subgraph selection, HetGCoT significantly enhances LLMs' ability to answer academic queries with interpretable explanations, outperforming existing state-of-the-art methods.
What Problem Does This Paper Address?
Academic question answering (QA) in scholarly networks is challenging because it requires understanding both the intricate relationships between entities (papers, authors, venues) and the semantic meaning of queries. Existing methods often struggle to holistically integrate these two aspects. While Graph Neural Networks (GNNs) can model graph structures, they lack the semantic comprehension of LLMs and often fail to adapt to specific queries or provide natural language explanations. LLMs, on the other hand, can understand text but cannot directly process the rich structural information embedded in academic networks. Current integration attempts are often superficial, treating graph information as mere auxiliary features rather than deeply embedding it into the reasoning process. This paper aims to bridge this gap by creating a framework that effectively leverages heterogeneous graph structures for interpretable academic QA.
Methods
HetGCoT is built upon three core technical contributions:
- Heterogeneous Graph to LLM Reasoning Chains: The framework transforms the complex structure of heterogeneous academic graphs into LLM-processable reasoning chains. This involves constructing a graph with nodes representing papers, authors, and venues, and edges representing relationships like "paper-venue" and "paper-author." Node features are initialized using Sentence-BERT embeddings for text and numerical attributes like citation counts. This graph is then encoded using a Heterogeneous Graph Transformer (HGT) to capture type-aware attention mechanisms.
- Adaptive Metapath Selection: To avoid processing the entire network, HetGCoT employs an adaptive mechanism to select relevant subgraphs. It leverages metapaths (sequences of relations) and learns their importance weights using FastGTN. This allows the framework to dynamically identify task-relevant subgraphs based on query semantics. A stratified selection strategy ensures diversity and quality of selected metapaths.
- Multi-step Reasoning Strategy: The framework integrates graph-derived contexts into a structured, four-step chain-of-thought (CoT) reasoning process within the LLM. This process involves:
- Graph Structure Analysis: Understanding structural patterns from naturalized metapaths.
- Content Analysis: Examining the target paper's information (title, abstract, keywords).
- Collaboration Analysis: Analyzing author collaboration patterns.
- Answer Generation: Synthesizing insights from previous steps to produce answers with explanations.
The metapaths are "naturalized" into natural language descriptions, prefixed with confidence scores, allowing LLMs to weigh structural evidence. The LLM is further enhanced through prompt engineering and task-specific fine-tuning.
Datasets
The framework is evaluated on two publicly available academic datasets:
- OpenAlex: A comprehensive, open index of scholarly works. For evaluation, a subgraph was extracted from journals ranked "A" or higher according to the CORE list.
- DBLP: A well-known computer science bibliography dataset. A subgraph was randomly sampled from the entire dataset.
The paper details the node types (paper, venue, author) and their attributes, as well as edge templates used for graph construction.
Key Insights and Ideas
- Bridging the Structure-Semantics Gap: The core innovation is the seamless integration of heterogeneous graph structures with LLM reasoning. Instead of treating them separately, HetGCoT creates a pipeline where graph information directly informs and guides the LLM's thought process.
- Interpretable Reasoning: The multi-step CoT approach, combined with naturalized metapaths, makes the reasoning process transparent. Users can understand why a particular answer is given, grounded in both structural evidence from the academic network and the content of the query.
- Adaptive Relevance: The adaptive metapath selection is crucial. It ensures that only the most relevant parts of the vast academic graph are considered for a given query, making the process more efficient and effective.
- Generalizability: The framework demonstrates adaptability across different LLM architectures and is effective for various academic QA tasks beyond journal recommendation, such as authorship identification and collaboration discovery.
Key Results
HetGCoT significantly outperforms all baseline methods on the OpenAlex and DBLP datasets for journal recommendation. For instance, on OpenAlex, it achieved a Hit rate of 96.48%, H@1 of 92.21%, and F1 score of 79.90%, surpassing even advanced graph+LLM integration methods. The framework also showed substantial improvements on authorship identification and collaboration discovery tasks. Ablation studies confirmed the importance of each component, particularly the CoT reasoning steps and the FastGTN-based metapath selection. The framework also showed consistent performance improvements when applied to different-sized LLMs, indicating its plug-and-play nature.
Limitations and Future Work
The authors acknowledge several limitations:
- LLM Output Instability: LLM outputs can exhibit some instability across different runs, especially for complex queries. Future work could explore more robust methods for ensuring consistent reasoning paths.
- Computational Requirements: Processing large heterogeneous academic graphs is computationally intensive, which might limit real-time applications without further optimization.
Future work could extend HetGCoT to more complex academic reasoning tasks, incorporate additional relationship types, and scale to larger, interdisciplinary datasets. The authors believe that combining structural graph information with LLM reasoning offers promising avenues for academic information processing.
Summary updated