TL;DR
This paper introduces HERCULES, a novel hyperbolic embedding model for temporal knowledge graphs (TKGs) that uniquely defines manifold curvature as a product of relation and time. Surprisingly, they find that even a non-temporal hyperbolic model (ATTH) achieves state-of-the-art results, suggesting that sophisticated temporal modeling might not be the sole driver of performance, and that effective negative sampling plays a crucial role.
What Problem Does This Paper Address?
Knowledge Graphs (KGs) represent factual information as triples (subject, predicate, object). Completing these graphs by predicting missing links is a vital task for many downstream applications like question answering and recommendation systems. However, traditional KG embedding methods often ignore the temporal aspect of facts, treating them as timeless. This is problematic because facts can become outdated or change over time (e.g., a person's political role). Temporal Knowledge Graphs (TKGs) address this by incorporating timestamps into facts, forming quadruplets (subject, predicate, object, timestamp). The challenge is to develop embedding models that can effectively capture both the relational structure and the temporal dynamics of these TKGs. Existing temporal KG embedding methods often extend static KG models by adding time as a parameter, but their effectiveness and the true contribution of temporal information remain areas of active research.
Methods
The paper builds upon the ATTH (Attentive Temporal Hyperbolic) model, which uses hyperbolic embeddings and learns relation-specific curvatures. The key innovation is HERCULES, a time-aware extension that defines the curvature of the hyperbolic manifold as the product of a learned relation parameter and a learned time parameter. This means the geometry of the embedding space dynamically adjusts based on both the relation and the specific timestamp.
Key methodological points:
- Hyperbolic Embeddings: The authors leverage hyperbolic geometry, specifically the Poincaré ball model, which is well-suited for hierarchical data structures often found in KGs.
- Relation-Specific Curvatures: ATTH already introduced trainable curvatures for each relation, allowing the geometry to adapt to different relation types.
- Time-Aware Curvature (HERCULES): HERCULES extends this by making the curvature dependent on time. The curvature $C_p$ for a relation $p$ is defined as $C_p = \sigma(\mu_p \times T_t)$, where $\mu_p$ is a relation-specific parameter and $T_t$ is a learnable time parameter. This allows the geometry to be influenced by both the relation and the timestamp.
- Negative Sampling: A significant finding is the impact of negative sampling. The authors demonstrate that optimizing ATTH with an increased number of negative samples (up to 500) leads to competitive or state-of-the-art results, even without explicit temporal modeling.
Datasets
The experiments were conducted on two standard TKG benchmark datasets:
- ICEWS14: Contains geopolitical events from 2014.
- ICEWS05-15: Encompasses geopolitical events from 2005 to 2015.
These datasets were originally constructed from the Integrated Crisis Early Warning System (ICEWS) dataset. The authors also employed a standard data augmentation technique by adding inverse relations ($p^{-1}$) to increase the number of training samples.
Key Insights and Ideas
- Time Might Not Be Everything (for performance): Astonishingly, the non-temporal ATTH model achieved state-of-the-art or comparable results to temporal models like HERCULES and DYERNIE. This suggests that the effectiveness of hyperbolic embeddings and careful optimization (especially negative sampling) might be more critical than explicit temporal modeling for achieving top performance on these datasets.
- The Power of Negative Sampling: Increasing the number of negative samples significantly boosted performance, leading to a rapid improvement in MRR up to around 200-300 samples. This indicates that a diverse set of negative examples helps constrain entities in the embedding space, leading to better relative positioning and thus improved link prediction.
- HERCULES's Subtle Geometry Adjustment: While HERCULES's time-dependent curvature doesn't dramatically outperform ATTH, it does subtly adjust the manifold's geometry. The paper shows that the learned geometries are generally similar between ATTH and HERCULES, with only rare steep discrepancies, suggesting that time's influence on curvature is often minor.
- Hyperbolic Geometry's Efficiency: Hyperbolic-based models (DYERNIE, ATTH, HERCULES) significantly outperform traditional models, especially at lower embedding dimensions, demonstrating the efficiency of hyperbolic geometry in capturing complex relational structures with fewer parameters.
Key Results
- State-of-the-Art Performance: Both ATTH and HERCULES achieved state-of-the-art or statistically equivalent performances on the ICEWS14 and ICEWS05-15 datasets, outperforming existing models.
- Dimensionality Efficiency: Hyperbolic models achieved strong results with significantly lower embedding dimensions (e.g., 10-40 dimensions) compared to traditional models (often requiring 100 dimensions).
- ATTH's Strong Baseline: ATTH, without explicit temporal modeling, achieved competitive results, sometimes even surpassing explicitly time-aware models.
- Time's Limited Impact: Experiments with corrupted timestamps showed that HERCULES's performance was largely insensitive to incorrect temporal information, further supporting the idea that time might not be the primary driver of performance gains in this context.
- Negative Sampling Impact: Increasing negative samples from 50 to 500 yielded an absolute gain of 5% points in MRR.
Limitations and Future Work
- Time's Role in Performance: The paper questions whether time is truly the key factor for performance gains in TKG link prediction, suggesting that other parameters (like negative sampling) might be more influential. This raises the question of when temporal information is truly necessary.
- Future Work: The authors plan to explore new mechanisms to incorporate temporal information to improve ATTH's performance. They also suggest that awareness should be raised when training TKG representations to identify if time is genuinely boosting performance.
- Optimization Challenges: While HERCULES's time-aware curvature is an interesting concept, the paper notes that optimizing hyperbolic embeddings can be challenging, and they resort to learning embeddings in Euclidean space and mapping them to the manifold.
Summary updated