TL;DR
The MORE framework (Metric learning-based Open Relation Extraction) shifts the focus of OpenRE from simple similarity classification to direct semantic representation learning. By combining Ranked List Loss with Virtual Adversarial Training, the authors achieve state-of-the-art results in clustering unseen relations, outperforming previous methods by significant margins on benchmark datasets.
What Problem Does This Paper Address?
In the world of Natural Language Processing, Relation Extraction (RE) is the task of identifying how two entities in a sentence are connected (e.g., in "London is the capital of the UK," the relation is capital_of). Traditional RE relies on a predefined set of labels. However, in the "open-domain" web, new relations emerge constantly.
Open Relation Extraction (OpenRE) aims to discover these relations without a fixed schema. Existing approaches usually fall into two camps:
- Tagging-based: These treat it as a sequence labeling task but often get stuck on surface-level word patterns.
- Clustering-based: These group similar sentences together to "discover" a relation.
The authors argue that current clustering-based methods are inefficient because they don't learn semantic representations directly. Instead, they often learn a "similarity classifier" (is A like B?), which is slow and doesn't necessarily create a clean, organized "feature space" where different relations are easily separable.
Methods
The authors propose MORE, a framework designed to map sentences into a high-dimensional space where sentences expressing the same relation are close together, and different relations are far apart.
1. Neural Encoders
The framework is flexible. The authors tested both a standard CNN and a pre-trained BERT model to convert sentences into initial vector representations. For BERT, they used a specific entity-marking technique to help the model focus on the relationship between the two entities in question.
2. Ranked List Loss (RLL)
This is the core innovation. Most metric learning uses "Triplet Loss" (comparing an anchor to one positive and one negative example). MORE uses Ranked List Loss, which looks at the entire batch of data.
- It pulls all positive instances (same relation) toward the anchor.
- It pushes all negative instances (different relations) beyond a specific "margin."
- Crucially, it weights "hard" negative examples (those that are confusingly close to the anchor) more heavily, providing a much richer supervision signal than simple pair-based losses.
3. Virtual Adversarial Training (VAT)
To make the model robust against the noise and diversity of open-domain text, the authors added VAT. This technique applies small, "worst-case" perturbations to the word embeddings during training. The model is penalized if these tiny changes cause a large jump in the resulting semantic representation, effectively "smoothing" the feature space.
Datasets
The model was evaluated on two major benchmarks:
- FewRel: A high-quality dataset derived from Wikipedia where entity pairs are unique, preventing the model from "cheating" by memorizing specific entities.
- NYT+FB-sup: A noisier dataset created via distant supervision from New York Times articles and Freebase. The authors specifically sampled this to create an "unbalanced" environment with many rare, novel relations to simulate real-world conditions.
Key Insights and Ideas
- Direct Representation is Key: Instead of asking "Are these two the same?", it is more efficient to ask "Where does this sentence live in relation-space?" By learning the space directly, downstream clustering (like K-Means) becomes much more accurate.
- The "Hypersphere" Advantage: Through t-SNE visualization (see Fig. 3 in the paper), the authors show that while previous methods (like RSNs) try to collapse relations into single points, MORE allows them to occupy "hyperspheres." This prevents the model from over-fragmenting a single relation into multiple tiny sub-clusters.
- Set-based vs. Point-based: Moving from triplet-based loss to set-based loss (RLL) allows the model to capture the global structure of the data much better, which is vital when you don't know how many categories you are looking for.
Key Results
The MORE framework established new state-of-the-art performance using the $B^3$ $F_1$ metric:
- On FewRel: MORE (using BERT) achieved an $F_1$ score of 74.5, a massive jump over the previous state-of-the-art (RSNs) at 59.9.
- On NYT+FB-sup: Even in a noisy, unbalanced environment, MORE (BERT) reached 49.7 $F_1$, compared to 38.8 for the previous best method.
- Ablation Study: Adding Virtual Adversarial Training (VAT) consistently improved the $F_1$ score for the CNN-based model, proving its worth in smoothing the semantic space.
Limitations and Future Work
- BERT + VAT: Interestingly, the authors did not apply VAT to the BERT-based version of MORE due to the "massive consumption of computing resources" required for fine-tuning BERT with adversarial perturbations. They noted this as a task for future work.
- Clustering Sensitivity: While the representations are better, the model still relies on downstream clustering algorithms (like K-Means or Mean-Shift). The choice of the "number of clusters" remains a classic challenge in OpenRE.
- Optimization Gap: The authors noted that VAT didn't provide a massive jump over RSNs in terms of "smoothing" the probability distribution space specifically, suggesting there is still room to optimize how adversarial training interacts with Euclidean distance metrics.
Summary updated