Abstract
Distributional embeddings capture general context but often lack precision for structured relationships in Knowledge Graphs (KGs). Traditional retrofitting assumes all graph edges imply similarity, which fails for complex relations like "treats" or "causes." We present Functional Retrofitting (FR), a framework that models relations as learnable functions. FR preserves semantic separation between disjoint entity classes while integrating complex graph knowledge, significantly improving performance on tasks like medical link prediction.
Introduction: Beyond Similarity
In Natural Language Processing, distributional embeddings (e.g., Word2Vec) represent word meanings based on context. While effective, these representations can be "blurry" regarding specific, structured relationships. Retrofitting is a common technique used to sharpen these embeddings by adjusting them so that entities connected in a Knowledge Graph (KG) are closer in vector space.
However, traditional retrofitting relies on a "similarity assumption"—the idea that every connection in a graph implies similarity. In complex graphs, this is often false. For example, if a graph states "Aragorn uses Athelas," a standard model might incorrectly force the embedding for a human king to be similar to that of a medicinal plant. This paper introduces Functional Retrofitting (FR), a framework that treats relations as functions rather than simple similarity constraints.
The Problem with the Similarity Assumption
The similarity assumption is a significant limitation for standard retrofitting. Most methods assume that if two nodes are connected, their vectors should be nearly identical. This fails in two major ways:
- Disjoint Entity Classes: In healthcare graphs, drugs and diseases are fundamentally different. Forcing them into the same neighborhood because a drug treats a disease distorts the semantic space.
- Diverse Relation Semantics: Relations can be complex. "Antonym of" implies the opposite of similarity, while "Causes" or "Uses" implies functional interaction rather than shared identity.
The authors propose a post-hoc method to adjust pre-trained embeddings using these complex relationships without losing the benefits of the original distributional data.
Methods: Functional Retrofitting (FR)
Instead of a simple distance penalty between connected nodes, FR introduces a relational penalty function $f_r$ for each relation type $r$.
- The Framework: The objective function balances three components: proximity to original distributional embeddings, satisfaction of functional relations in the graph, and a "negative space" penalty to prevent vector collapse.
- Linear Instantiation: Relations are modeled as linear transformations: $q_i \approx A_r q_j + b_r$. This allows the model to "shift" or "rotate" an entity's meaning into the space of its neighbor.
- Neural Instantiation: A version using a Neural Tensor Network (NTN) approach with tanh nonlinearities captures more complex interactions.
- Optimization: For the linear case, closed-form solutions for block optimization make the process highly efficient for large-scale graphs.
Key Insights: Preserving Semantic Separation
A core innovation is treating a graph edge as an operator that transforms one point toward another, rather than "glue" pulling them together. This allows the model to learn that a drug is related to a disease without making the drug vector look like a disease vector.
As shown in the t-SNE visualizations, FR preserves distinct clusters (e.g., drugs vs. diseases) while standard identity retrofitting blurs them. Negative sampling is also crucial; the model must see what a relation isn't to learn its true meaning.
Key Results
- Superior Link Prediction: On FrameNet's "Causative Of" relation, the neural FR model achieved 94.33% accuracy, compared to 84.49% for standard retrofitting.
- Healthcare Utility: In the Roam Health Knowledge Graph, FR-Linear reached 84.22% accuracy in predicting "Treats" relations, a significant improvement over the 72.93% achieved by standard methods.
- Drug Retargeting: The model successfully predicted plausible medical treatments not in the original graph, such as Kenalog for contact dermatitis, a claim supported by clinical trials.
- No Regression: On WordNet, where similarity holds, FR performed as well as or slightly better than baselines, proving it is a safe generalization.
Limitations and Future Work
Current limitations include the manual setting of weights between distributional and graph data ($\alpha$ and $\beta$). Future work may involve learning these weights dynamically. Additionally, while the linear solution scales well, the neural version requires more intensive optimization. Exploring better modeling for bi-directional or symmetric relations within this functional framework remains an open research question.
Summary updated