TL;DR
This paper introduces "Domain Frequency" (df), a novel term weighting scheme for Open Information Extraction (Open IE) that significantly improves relation extraction accuracy. By prioritizing terms that are specific to particular entity type domains (e.g., "wife" for PER-PER relations), it overcomes limitations of traditional TF-IDF, leading to more robust clustering of entity pairs into meaningful relations.
What Problem Does This Paper Address?
Open Information Extraction (Open IE) aims to extract all possible relations between named entities from unstructured text. A common approach involves clustering entity pairs based on the textual context in which they appear. However, existing systems often struggle with noisy data and identifying truly discriminative relational terms. Traditional TF-IDF weighting, while useful, can penalize terms that are highly indicative of a specific relation but appear frequently within that relation's domain, making it harder to cluster them correctly. This paper tackles the challenge of improving the accuracy of these clustering-based Open IE systems by developing a more effective term weighting scheme.
Methods
The core contribution of this paper is the Domain Frequency (df) weighting scheme. Unlike TF-IDF, which measures a term's rarity across the entire corpus, df measures a term's frequency within specific relation domains (defined by pairs of entity types, e.g., PER-PER for person-person relations). The formula for df is:
$df_i(t) = \frac{f_i(t)}{\sum_{j<n} f_j(t)}$
where $f_i(t)$ is the frequency of term $t$ in relation domain $i$, and $n$ is the total number of relation domains. This scheme aims to boost the weight of terms that are highly specific to a particular relation type.
The authors integrate this df score in two ways:
- As a term weight: They use the product of TF, IDF, and DF ($tf \cdot idf \cdot df$) to weight terms in the context vectors of entity pairs.
- As a pruning filter: They use minimum df and maximum idf thresholds to remove noisy terms from the context vectors.
The system is built upon the SONEX Open IE system and uses Hierarchical Agglomerative Clustering (HAC) with average link.
Datasets
The evaluation was performed on two datasets:
- INTER: Contains intersection pairs from the Spinn3r corpus and Freebase, matching both entity names and types exactly. It includes 395 entity pairs and 20 relations.
- NOISY: Derived from INTER, it includes approximately 30,000 entity pairs, making it a larger and more challenging benchmark.
These datasets were extended from previous work by Mesquita et al.
Key Insights and Ideas
- Domain Specificity Matters: The most surprising insight is how much more effective domain-specific weighting is compared to global weighting (IDF). Terms like "wife" are highly indicative of PER-PER relations, and df correctly amplifies their importance in that context.
- IDF's Limitation in Open IE: The paper highlights that for Open IE, high IDF doesn't necessarily mean a term is a good relation discriminator. A term might be rare globally but still not specific to a particular relation.
- DF as a Complement to NER: The df score can help identify relational patterns associated with specific entity types, potentially aiding Named Entity Recognition (NER) systems, especially in domains where NER performance is weaker.
- Pruning is Crucial: Using df as a filter, in conjunction with maximum idf filtering, significantly reduces noise and further improves clustering accuracy.
Key Results
- Integrating the df score ($tf \cdot idf \cdot df$) significantly outperformed the baseline ($tf \cdot idf$) on both INTER and NOISY datasets.
- INTER: F-1 score improved from 0.75 to 0.87.
- NOISY: F-1 score improved from 0.65 to 0.72.
- Applying minimum df and maximum idf filtering further improved results on the INTER dataset.
- The results consistently dropped for clustering thresholds larger than 0.05, indicating sensitivity to parameter tuning.
- The NOISY dataset showed a substantial decrease in performance compared to INTER, underscoring the challenge of scaling Open IE to larger, more complex datasets with more relations.
Limitations and Future Work
- NER Dependency: The performance of the df score is still dependent on the accuracy of the underlying Named Entity Recognition (NER) system. Errors in NER can lead to misleading domain assignments.
- Language Independence: While the df score is largely language-independent due to its probabilistic nature, some adjustments might be needed for languages with different syntactic structures (e.g., German, where relational verbs might not always appear between entities).
- Future Work: The authors suggest extending df to a multi-lingual environment and exploring its application in improving NER systems by identifying relational patterns. They also note that the list of terms and scores generated by df can be used "off-the-shelf" for various applications.
Summary updated