Abstract
Training Relational Graph Convolutional Networks (R-GCN) on large-scale knowledge graphs is often constrained by the "memory wall"—the excessive GPU memory required to store gradient information during backpropagation. This research investigates the use of graph summarization techniques to compress input graphs, enabling more efficient training. By training an R-GCN on a condensed summary graph and transferring the learned weights back to the original graph, we achieve comparable or superior accuracy across several standard datasets (AIFB, MUTAG, and AM). Our results demonstrate that graph summarization not only reduces computational overhead but can also act as a regularizer, improving model performance on specific tasks.
Introduction
Knowledge Graphs (KGs) represent complex relationships between entities and serve as a foundation for various AI applications, from recommendation engines to semantic search. Relational Graph Convolutional Networks (R-GCNs) are specifically designed to handle the multi-relational nature of these graphs. However, the memory intensity of R-GCNs scales with the size and complexity of the graph, often exceeding the capacity of modern hardware. While traditional scaling solutions focus on hardware expansion or algorithmic optimization, this work explores a data-centric compression strategy: reducing the graph's size while preserving the structural "essence" required for learning.
Methodology: The Summarize-then-Transfer Framework
The core of this research is a two-stage pipeline designed to bypass the memory limitations of full-graph training.
1. Graph Summarization
The authors utilize graph summarization algorithms to group "equivalent" nodes into partitions, creating a smaller, representative summary graph. Two primary methods were evaluated:
- Attributes Summary (Approximate): This method groups nodes based on their local neighborhood, specifically those sharing the same set of outgoing edge labels (attributes).
- $k$-forward Bisimulation (Precise): A more rigorous structural approach that groups nodes only if they share identical paths over $k$ hops. This ensures a higher degree of structural fidelity in the summary.
2. Weighted Multi-label Training
In a summary graph, a single node represents a cluster of original nodes. Because these original nodes may belong to different classes, the authors implement a weighted multi-label training strategy. The target label for a summary node is determined by the frequency distribution of the classes within its constituent nodes, ensuring the model learns the dominant characteristics of each partition.
3. Weight Transfer and Inference
Once the R-GCN is trained on the summary graph, its weights are transferred to a model configured for the original graph. This "transfer model" can be used for immediate inference or further fine-tuned. This approach allows the model to benefit from the structural patterns learned during the efficient summary-training phase, providing a significant head start in performance.
Experimental Evaluation
The framework was tested on three RDF (Resource Description Framework) datasets of varying scales:
- AIFB: A small validation dataset with 8,285 entities.
- MUTAG: A medium-sized dataset (23,644 entities) featuring 113 distinct classes, providing a complex classification challenge.
- AM: A large-scale dataset containing 1.66 million entities, representing real-world data volumes and scaling requirements.
Key Insights and Results
The "Jump-Start" Effect
One of the most significant findings was the "jump-start" provided by weight transfer. Models initialized with weights from summary training consistently exhibited high accuracy (often 70-80%) at Epoch 0 on the original graph. This suggests that the summary graph successfully captures the fundamental relational patterns necessary for the task.
Summarization as a Regularizer
In the AIFB dataset, the transfer model reached an accuracy of 92.5%, notably outperforming the 87.9% achieved by the baseline model trained on the full graph. This indicates that summarization may act as a form of regularization, filtering out "noise" and forcing the model to focus on the core schema and high-level structural features of the graph.
Efficiency and Trade-offs
On the large-scale AM dataset, the transfer model achieved results comparable to the baseline but reached high accuracy much earlier in the training process. However, the researchers noted a trade-off between conciseness and precision. In the MUTAG dataset, very aggressive summarization led to a loss of vital information, allowing the baseline model to eventually catch up. Additionally, the inclusion of "literal" nodes (strings or dates) was found to be critical; removing them significantly degraded the performance of the transfer models.
Conclusion and Future Work
This study demonstrates that graph summarization is a viable strategy for scaling R-GCN training. By reducing the graph size, we can significantly lower the memory footprint and computational cost per epoch. While the initial cost of computing the summary is a factor that must be managed, the gains in training efficiency and the potential for improved generalization make this a promising direction for large-scale graph learning. Future research will focus on optimizing summarization algorithms to reduce pre-processing time and exploring the effectiveness of this approach for link prediction tasks.
Summary updated