LGA: LLM-GNN Aggregation for Temporal Evolution Attribute Graph Prediction

Feng Zhao, Ruoyu Chai, Kangzheng Liu, Xianggan LiuEMNLP 2025

TL;DR

This paper introduces LGA, a novel framework that unifies Large Language Models (LLMs) and Graph Neural Networks (GNNs) for predicting how node attributes evolve over time in dynamic graphs. By enabling LLMs to perform GNN-like aggregation and by enriching attribute embeddings with structural semantics, LGA significantly improves prediction accuracy, addressing limitations of existing isolated LLM and GNN approaches.

What Problem Does This Paper Address?

Predicting the future attributes of entities in dynamic systems is crucial for many real-world applications, from fault detection to economic forecasting. These systems are often modeled as "temporal evolution attribute graphs," where entities (nodes) interact over time, and their attributes change. While recent advancements have seen LLMs used to enhance node representations for GNNs, their potential for direct, integrated aggregation within GNN frameworks remains underexplored. Furthermore, traditional methods often initialize attribute embeddings without considering the graph's structural information, leading to weak semantic representations that hinder downstream tasks. Existing GNN approaches also tend to focus only on immediate neighbors, missing complex, long-range structural dependencies. This paper aims to bridge these gaps by developing a unified LLM-GNN approach that leverages LLMs for sophisticated aggregation and incorporates structural semantics into attribute embeddings.

Methods

The proposed LGA framework consists of three main stages:

  1. Initialization Embedding Stage: This stage focuses on creating richer initial attribute embeddings. A novel "attribute embedding loss" is introduced, inspired by existing work but modified to align the embedding space with the topological space. This helps reduce the distance between embeddings of nodes that are close in the graph structure, effectively incorporating topological semantics. A distance-based function is used to assign weights to links based on hop count, compensating for regularization gaps.
  2. Neighborhood Aggregation Stage: This is the core innovation. LGA designs specialized prompts to enable LLMs to perform GNN-like aggregation. These prompts guide the LLM to understand node neighborhood information in terms of links and prioritize neighbors based on attribute differences. To capture complex, long-range structural interactions, a relation-aware Graph Convolutional Network (R-GCN) module is incorporated. This R-GCN module separately processes node and attribute embeddings to capture multi-hop information propagation.
  3. Prediction Stage: The aggregated information from both the LLM and R-GCN modules is concatenated and fed into a Gated Recurrent Unit (GRU) to capture temporal dependencies and generate future attribute predictions. The framework also includes mechanisms for predicting future graph structures.

Datasets

The authors evaluated LGA on five real-world, open-source datasets from diverse domains:

  • CAC: Coauthorship-Citation dataset, representing author citation information.
  • MTG: Multiattributed Trade Graph, representing international trade situations.
  • NBA: Represents team formation status between players.
  • AGG: Attributed GDELT Graph, representing global events, language, and tone.
  • ATG: Attributed Trade Graph, representing export trade between countries.

These datasets vary in graph complexity, number of node attributes, and relationship types, providing a comprehensive evaluation.

Key Insights and Ideas

  • LLMs as GNN Aggregators: The paper demonstrates that LLMs can be prompted to perform GNN-like aggregation, effectively combining their external knowledge with graph structural information. This moves beyond simply using LLMs for initial feature enhancement.
  • Structural Semantics for Embeddings: Incorporating topological information directly into attribute embeddings through a specialized loss function significantly improves their semantic richness, making nodes more distinguishable and aiding downstream prediction.
  • Long-Range Dependencies Matter: The R-GCN module's ability to capture multi-hop interactions is crucial for understanding complex graph structures, leading to better prediction accuracy than methods limited to 1-hop neighborhoods.
  • Unified Framework Benefits: Combining LLM-driven aggregation with GNN-based structural capture in a single framework yields superior performance compared to treating these components in isolation.

Key Results

LGA consistently outperforms baseline methods across the evaluated datasets. For instance, in temporal evolution attribute graph prediction, LGA achieves the best results on most datasets, with bold formatting indicating the top performance. Specifically, on the CAC dataset, LGA achieved an MSE loss of 1.934, significantly outperforming other models. The paper also shows that LGA's approach to attribute embedding leads to a substantial reduction in Euclidean distances between node representations and their neighbors, indicating improved feature similarity. Ablation studies confirm that each component of LGA (LLM aggregation, attribute embedding loss, and long-distance capture) contributes significantly to the overall performance.

Limitations and Future Work

The authors acknowledge that their model's reliance on GNN modules for information aggregation increases training time and GPU memory usage, especially for larger graphs. The embedding stage's training time also accelerates with graph complexity. As historical time length increases, more GPU memory is required. Future work will focus on adapting LGA for larger-scale attribute graphs while minimizing computational resource consumption.

Summary updated

BibTeX
@inproceedings{zhao2025lga,
  author = {Feng Zhao and Ruoyu Chai and Kangzheng Liu and Xianggan Liu},
  title = {{LGA: LLM-GNN Aggregation for Temporal Evolution Attribute Graph Prediction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure