ECNU: Effective Semantic Relations Classification without Complicated Features or Multiple External Corpora

Yuan Chen, Man Lan, Jian Su, Zhi-Min Zhou, Yu XuSemEval @ ACL 2010 · OpenReview

Abstract

Yuan Chen, Man Lan, Jian Su, Zhi Min Zhou, Yu Xu. Proceedings of the 5th International Workshop on Semantic Evaluation.

2010.

Introduction

Identifying semantic relationships between nominals in natural language—such as "The apple is in the basket" (Content-Container)—is a core challenge for information extraction and question answering. Historically, high performance on this task required complex systems leveraging massive external datasets and intricate feature engineering. Researchers from East China Normal University and the Institute for Infocomm Research proposed a more streamlined, "practical" approach. Their goal was to demonstrate that a system using minimal external resources and a unified feature set could achieve competitive results on the SemEval-2010 Task 8 benchmark.

Methodology

The ECNU system utilizes Support Vector Machines (SVMs) with a global feature set categorized into six primary groups:

  1. Lexical Features: Lemmas of the target nouns.
  2. Morpho-syntactic Features: Stems and Part-of-Speech (POS) tags of intervening words.
  3. Syntactic Patterns: Structural context derived from the "parent" node in a grammatical parse tree.
  4. WordNet Semantic Classes: High-level categories (e.g., "entity", "act") for the nouns.
  5. Meronym-Holonym Relations: Specific WordNet checks for "part-of" relationships to assist with Component-Whole and Member-Collection relations.
  6. Container Check: A specialized binary feature identifying "container" hypernyms for the Content-Container relation.

The researchers compared two classification strategies: a single 10-way classifier and a multiple binary classification approach (one for each of the 9 relations).

Experimental Setup

The system was evaluated using the SemEval-2010 Task 8 dataset, comprising 8,000 training and 2,717 test sentences across 9 distinct relations and one "Other" category.

Results and Analysis

The team submitted seven configurations (SR1–SR7). The results demonstrated that incorporating WordNet features significantly boosted performance, with F1-measures rising from approximately 60% to 75%.

Performance comparison of different system configurations (SR1-SR7).

As shown in the performance table, the multiple binary classification strategy generally outperformed the multi-way classifier. Specifically, Run SR5—which used all six feature sets, binary SVMs, and a probability threshold of 0.25—achieved the highest F1-measure of 75.43%. The lower threshold improved recall, leading to a better overall balance.

Detailed performance breakdown by relation type for the best-performing configuration (SR5).

Performance varied significantly across relation types. The system was most effective at identifying Cause-Effect (85.07% F1) and Entity-Destination (83.01% F1). Conversely, it struggled with the Instrument-Agency relation (60.67% F1), suggesting that certain categories may require more specialized features.

Conclusion

The ECNU system proves that effective semantic relation classification is achievable without excessive complexity. While ensemble methods (SR7) did not provide a significant boost due to model similarity, the strategic use of WordNet and binary classification proved highly effective. Future work may explore more sophisticated Word Sense Disambiguation and diverse classifier ensembles to further improve performance on challenging relations.

Summary updated

BibTeX
@inproceedings{chen2010ecnu,
  author = {Yuan Chen and Man Lan and Jian Su and Zhi-Min Zhou and Yu Xu},
  title = {{ECNU: Effective Semantic Relations Classification without Complicated Features or Multiple External Corpora}},
  booktitle = {SemEval @ ACL},
  year = {2010},
  url = {https://openreview.net/forum?id=Sc8VFa80Afc},
}
Figure