Abstract
Yuan Chen, Man Lan, Jian Su, Zhi Min Zhou, Yu Xu. Proceedings of the 5th International Workshop on Semantic Evaluation.
2010.
Introduction
Identifying semantic relationships between nominals in natural language—such as "The apple is in the basket" (Content-Container)—is a core challenge for information extraction and question answering. Historically, high performance on this task required complex systems leveraging massive external datasets and intricate feature engineering. Researchers from East China Normal University and the Institute for Infocomm Research proposed a more streamlined, "practical" approach. Their goal was to demonstrate that a system using minimal external resources and a unified feature set could achieve competitive results on the SemEval-2010 Task 8 benchmark.
Methodology
The ECNU system utilizes Support Vector Machines (SVMs) with a global feature set categorized into six primary groups:
- Lexical Features: Lemmas of the target nouns.
- Morpho-syntactic Features: Stems and Part-of-Speech (POS) tags of intervening words.
- Syntactic Patterns: Structural context derived from the "parent" node in a grammatical parse tree.
- WordNet Semantic Classes: High-level categories (e.g., "entity", "act") for the nouns.
- Meronym-Holonym Relations: Specific WordNet checks for "part-of" relationships to assist with Component-Whole and Member-Collection relations.
- Container Check: A specialized binary feature identifying "container" hypernyms for the Content-Container relation.
The researchers compared two classification strategies: a single 10-way classifier and a multiple binary classification approach (one for each of the 9 relations).
Experimental Setup
The system was evaluated using the SemEval-2010 Task 8 dataset, comprising 8,000 training and 2,717 test sentences across 9 distinct relations and one "Other" category.
Results and Analysis
The team submitted seven configurations (SR1–SR7). The results demonstrated that incorporating WordNet features significantly boosted performance, with F1-measures rising from approximately 60% to 75%.
As shown in the performance table, the multiple binary classification strategy generally outperformed the multi-way classifier. Specifically, Run SR5—which used all six feature sets, binary SVMs, and a probability threshold of 0.25—achieved the highest F1-measure of 75.43%. The lower threshold improved recall, leading to a better overall balance.
Performance varied significantly across relation types. The system was most effective at identifying Cause-Effect (85.07% F1) and Entity-Destination (83.01% F1). Conversely, it struggled with the Instrument-Agency relation (60.67% F1), suggesting that certain categories may require more specialized features.
Conclusion
The ECNU system proves that effective semantic relation classification is achievable without excessive complexity. While ensemble methods (SR7) did not provide a significant boost due to model similarity, the strategic use of WordNet and binary classification proved highly effective. Future work may explore more sophisticated Word Sense Disambiguation and diverse classifier ensembles to further improve performance on challenging relations.
Summary updated