Introduction
Relation Extraction (RE) is a fundamental task in Natural Language Processing, focused on identifying semantic relationships between entities within a text—for instance, determining that "Mary" is the parent of "Jerry" in the sentence "Mary gave birth to Jerry." While modern RE models have achieved high performance using transformer-based architectures like BERT and LUKE, they typically treat sentences as isolated units. This "local" approach often fails when a single sentence lacks sufficient context to disambiguate a relation.
Researchers have long recognized that "global" information—patterns of entity behavior and topical context across an entire dataset—could provide the necessary signals to improve accuracy. However, integrating this global context using standard Graph Neural Networks (GNNs) is computationally prohibitive. Standard GNNs require processing connections across the entire dataset at every training step, leading to memory bottlenecks and high latency. GRAPHCACHE addresses this challenge by framing GNN message passing as a caching mechanism, enabling the use of global features with almost zero additional computational overhead.
The GRAPHCACHE Framework
GRAPHCACHE is designed as a modular component that can be integrated into existing RE backbones. Its architecture is built on three primary pillars:
1. Heterogeneous Graph Construction
Instead of a simple sentence-to-sentence graph, the authors construct a heterogeneous graph where sentences are linked to "property nodes." These nodes represent two types of global priors:
- Entity Types: Categorical information (e.g., Person-Date or Organization-Location).
- Contextual Topics: Latent themes mined from the dataset using Latent Dirichlet Allocation (LDA), capturing the general subject matter of the sentence.
2. Caching Message Passing
The core innovation of GRAPHCACHE is the analogy between GNN updates and computer system caching. In a standard GNN, updating a node requires aggregating information from its neighbors across the entire graph ($O(N)$ complexity). GRAPHCACHE treats property nodes as caches that store aggregated sentence representations.
During training, the model employs an online updating strategy: as a batch of sentences is processed, the model only updates the specific property caches connected to those sentences. This reduces the complexity from $O(N)$ to $O(B)$, where $B$ is the batch size. This ensures that global information remains "fresh" and evolves alongside the model weights without requiring a full graph traversal.
3. Global-Local Fusion
To make a final prediction, the model uses an attention mechanism to fuse the sentence’s local features (from the transformer backbone) with the global features retrieved from the property caches. This allows the model to "denoise" ambiguous local signals using dataset-level prior knowledge.
Experimental Evaluation
The authors evaluated GRAPHCACHE on three prominent RE benchmarks:
- TACRED: A large-scale dataset containing over 106,000 mention pairs.
- TACREV: A refined version of TACRED with corrected labels for more reliable evaluation.
- SemEval 2010 Task 8: A classic multi-way classification dataset for semantic relations.
Performance and Efficiency
GRAPHCACHE consistently improved the performance of strong baselines. Notably, it boosted the LUKE model by 2.9% F1 on the TACREV dataset and 1.5% on SemEval.
Crucially, these gains did not come at the cost of efficiency. On the TACRED dataset, the GRAPHCACHE-enhanced version of IRE-RoBERTa took 7681 seconds to train, compared to 7492 seconds for the original model. This negligible increase in training time demonstrates that the caching strategy effectively bypasses the traditional scaling issues associated with GNNs.
Key Insights
- Robustness to Unseen Entities: GRAPHCACHE performs exceptionally well on entities that did not appear in the training set. By learning the general behavior of entity types and topics across the dataset, the model becomes less dependent on memorizing specific entity names.
- The Power of Context: Ablation studies revealed that contextual topics (the "vibe" of the sentence) often provided a more significant performance boost than entity types alone, suggesting that topical priors are highly effective at disambiguating relations.
- Theoretical Soundness: The authors provide mathematical proof (Proposition 1) showing that their caching approach is functionally equivalent to a standard GNN layer while maintaining significantly lower computational requirements.
Conclusion and Future Work
GRAPHCACHE demonstrates that global dataset information can be harnessed efficiently for sentence-level tasks. By treating message passing as a caching problem, the model achieves state-of-the-art results without the typical memory and latency penalties of GNNs.
Future research directions include moving away from static preprocessing (like LDA) toward end-to-end learnable topic models and extending the caching logic to Document-Level Relation Extraction, where entities and relations are distributed across multiple sentences and require even more sophisticated context management.
Summary updated