A Comparative Study on Statistical Classification Methods in Relation Extraction

Xiaofeng Zhang, Zhiqiang Gao, Yaocheng GuiAdvances 2013 · DOI

Abstract

This paper is a comparative study of statistical classification approaches in relation extraction and classification. The focus is on multiclass classification, not on sequence labeling. Five methods are evaluated, including naive Bayes (NB), decision tree (DT), k-nearest neighbor (kNN), support vector machine (SVM) and sparse network of Winnow (SNoW). Using DT on Roth and Yih data set, the best precision and recall are achieved on both tasks of named entity recognition (NER) and relation extraction (RE). SNoW is not so good as DT, but it performs better than the other approaches. SVM performs better on precision and worse on recall. In contrast, the simplest methods NB and kNN has relative poor performance but they are not sensitive to learning tasks and classes.

Introduction

Information Extraction (IE) is a critical component in converting unstructured text into structured data. This process generally involves two primary tasks: Named Entity Recognition (NER), which identifies entities like persons or locations, and Relation Extraction (RE), which identifies the semantic connections between them (e.g., "Work_For" or "Located_In").

While much research focuses on engineering advanced linguistic features, such as dependency trees or external knowledge bases, there is a lack of direct comparison between the underlying statistical algorithms when features are standardized. This study evaluates five classic classifiers to determine which is inherently best suited for the imbalanced and often noisy nature of relational data.

Methodology

The researchers evaluated five distinct classification strategies:

  1. Decision Tree (DT): Specifically the J48 implementation of the C4.5 algorithm.
  2. Support Vector Machine (SVM): Using the WLSVM implementation.
  3. Sparse Network of Winnow (SNoW): A multi-class classifier tailored for large feature spaces.
  4. Naive Bayes (NB): A generative model based on conditional independence.
  5. k-Nearest Neighbor (kNN): A non-parametric approach using Euclidean distance.

To isolate algorithmic performance, the study intentionally avoided advanced linguistic features, using 30 standard features for NER (e.g., word forms, POS tags) and 100 features for RE (e.g., keyword patterns and word distance). The study utilized the Roth and Yih dataset, which presents a significant challenge due to the "No Relation" (NR) class outnumbering actual relations.

Key Results

Named Entity Recognition (NER)

In the NER task, the Decision Tree (DT) achieved the highest overall performance. For instance, it reached an F1-score of 82.2 for identifying Persons (PER), with SNoW following closely at 81.7. The simpler methods, NB and kNN, showed lower but more stable performance across different entity types.

Performance comparison of five classification methods on Named Entity Recognition (NER) tasks.

Relation Extraction (RE)

The RE task proved more difficult, but the Decision Tree remained the top performer, achieving an F1 of 51.9 for the LOCATED_IN relation. A notable finding was the performance of SVMs; while they provided high precision, they suffered from extremely low recall, often missing a large portion of valid relations. For example, in the LOCATED_IN category, SVM achieved a precision of 73.4 but a recall of only 11.7.

Performance comparison of five classification methods on Relation Extraction (RE) tasks.

Discussion and Conclusion

The study demonstrates that the hierarchical, rule-based nature of Decision Trees is particularly effective for the standard feature sets used in IE. SVMs, despite their popularity in high-dimensional spaces, struggled with recall in this specific domain. SNoW emerged as a strong and robust alternative for multi-class classification.

While absolute scores were lower than state-of-the-art systems due to the exclusion of advanced features, this experimental design successfully isolated the inherent strengths of each algorithm. Future work may explore if richer linguistic features allow complex models like SVMs to overcome their recall limitations.

Summary updated

Abstract

This paper is a comparative study of statistical classification approaches in relation extraction and classification.The focus is on multiclass classification, not on sequence labeling.Five methods are evaluated, including naive Bayes (NB), decision tree (DT), k-nearest neighbor (kNN), support vector machine (SVM) and sparse network of Winnow (SNoW).Using DT on Roth and Yih data set, the best precision and recall are achieved on both tasks of named entity recognition (NER) and relation extraction (RE).SNoW is not so good as DT, but it performs better than the other approaches.SVM performs better on precision and worse on recall.In contrast, the simplest methods NB and kNN has relative poor performance but they are not sensitive to learning tasks and classes.
BibTeX
@inproceedings{zhang2013comparative,
  author = {Xiaofeng Zhang and Zhiqiang Gao and Yaocheng Gui},
  title = {{A Comparative Study on Statistical Classification Methods in Relation Extraction}},
  booktitle = {Advances},
  year = {2013},
  doi = {10.2991/icmt-13.2013.21},
}
Figure