GraphCache: Message Passing as Caching for Sentence-Level Relation Extraction

Yiwei Wang, Muhao Chen, Wenxuan Zhou, Yujun Cai, Yuxuan Liang, Bryan Hooi

Introduction

Relation Extraction (RE) is a fundamental task in Natural Language Processing, focused on identifying semantic relationships between entities within a text—for instance, determining that "Mary" is the parent of "Jerry" in the sentence "Mary gave birth to Jerry." While modern RE models have achieved high performance using transformer-based architectures like BERT and LUKE, they typically treat sentences as isolated units. This "local" approach often fails when a single sentence lacks sufficient context to disambiguate a relation.

Researchers have long recognized that "global" information—patterns of entity behavior and topical context across an entire dataset—could provide the necessary signals to improve accuracy. However, integrating this global context using standard Graph Neural Networks (GNNs) is computationally prohibitive. Standard GNNs require processing connections across the entire dataset at every training step, leading to memory bottlenecks and high latency. GRAPHCACHE addresses this challenge by framing GNN message passing as a caching mechanism, enabling the use of global features with almost zero additional computational overhead.

The GRAPHCACHE Framework

GRAPHCACHE is designed as a modular component that can be integrated into existing RE backbones. Its architecture is built on three primary pillars:

1. Heterogeneous Graph Construction

Instead of a simple sentence-to-sentence graph, the authors construct a heterogeneous graph where sentences are linked to "property nodes." These nodes represent two types of global priors:

  • Entity Types: Categorical information (e.g., Person-Date or Organization-Location).
  • Contextual Topics: Latent themes mined from the dataset using Latent Dirichlet Allocation (LDA), capturing the general subject matter of the sentence.

2. Caching Message Passing

The core innovation of GRAPHCACHE is the analogy between GNN updates and computer system caching. In a standard GNN, updating a node requires aggregating information from its neighbors across the entire graph ($O(N)$ complexity). GRAPHCACHE treats property nodes as caches that store aggregated sentence representations.

During training, the model employs an online updating strategy: as a batch of sentences is processed, the model only updates the specific property caches connected to those sentences. This reduces the complexity from $O(N)$ to $O(B)$, where $B$ is the batch size. This ensures that global information remains "fresh" and evolves alongside the model weights without requiring a full graph traversal.

3. Global-Local Fusion

To make a final prediction, the model uses an attention mechanism to fuse the sentence’s local features (from the transformer backbone) with the global features retrieved from the property caches. This allows the model to "denoise" ambiguous local signals using dataset-level prior knowledge.

Experimental Evaluation

The authors evaluated GRAPHCACHE on three prominent RE benchmarks:

  • TACRED: A large-scale dataset containing over 106,000 mention pairs.
  • TACREV: A refined version of TACRED with corrected labels for more reliable evaluation.
  • SemEval 2010 Task 8: A classic multi-way classification dataset for semantic relations.

Performance and Efficiency

GRAPHCACHE consistently improved the performance of strong baselines. Notably, it boosted the LUKE model by 2.9% F1 on the TACREV dataset and 1.5% on SemEval.

Crucially, these gains did not come at the cost of efficiency. On the TACRED dataset, the GRAPHCACHE-enhanced version of IRE-RoBERTa took 7681 seconds to train, compared to 7492 seconds for the original model. This negligible increase in training time demonstrates that the caching strategy effectively bypasses the traditional scaling issues associated with GNNs.

Key Insights

  • Robustness to Unseen Entities: GRAPHCACHE performs exceptionally well on entities that did not appear in the training set. By learning the general behavior of entity types and topics across the dataset, the model becomes less dependent on memorizing specific entity names.
  • The Power of Context: Ablation studies revealed that contextual topics (the "vibe" of the sentence) often provided a more significant performance boost than entity types alone, suggesting that topical priors are highly effective at disambiguating relations.
  • Theoretical Soundness: The authors provide mathematical proof (Proposition 1) showing that their caching approach is functionally equivalent to a standard GNN layer while maintaining significantly lower computational requirements.

Conclusion and Future Work

GRAPHCACHE demonstrates that global dataset information can be harnessed efficiently for sentence-level tasks. By treating message passing as a caching problem, the model achieves state-of-the-art results without the typical memory and latency penalties of GNNs.

Future research directions include moving away from static preprocessing (like LDA) toward end-to-end learnable topic models and extending the caching logic to Document-Level Relation Extraction, where entities and relations are distributed across multiple sentences and require even more sophisticated context management.

Summary updated

Abstract

Entity types and textual context are essential properties for sentence-level relation extraction (RE). Existing work only encodes these properties within individual instances, which limits the performance of RE given the insufficient features in a single sentence. In contrast, we model these properties from the whole dataset and use the dataset-level information to enrich the semantics of every instance. We propose the GRAPHCACHE (Graph Neural Network as Caching) module, that propagates the features across sentences to learn better representations for RE. GRAPHCACHE aggregates the features from sentences in the whole dataset to learn global representations of properties, and use them to augment the local features within individual sentences. The global property features act as dataset-level prior knowledge for RE, and a complement to the sentence-level features. Inspired by the classical caching technique in computer systems, we develop GRAPHCACHE to update the property representations in an online manner. Overall, GRAPHCACHE yields significant effectiveness gains on RE and enables efficient message passing across all sentences in the dataset.
BibTeX
@inproceedings{wang2022graphcache,
  author = {Yiwei Wang and Muhao Chen and Wenxuan Zhou and Yujun Cai and Yuxuan Liang and Bryan Hooi},
  title = {{GraphCache: Message Passing as Caching for Sentence-Level Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.128},
}
Figure