A Java Framework for Multilingual Definition and Hypernym Extraction

Stefano Faralli, Roberto NavigliACL 2013

TL;DR

This paper introduces a Java framework and web application for automatically extracting definitions and hypernyms from text, extending previous work to English, French, and Italian. By developing a novel, automated method for creating multilingual training data from Wikipedia, they demonstrate robust definition and hypernym extraction across languages, making this powerful NLP capability more accessible.

What Problem Does This Paper Address?

Electronic dictionaries and domain glossaries are invaluable resources, but their creation and maintenance are labor-intensive. While raw text often contains definitional sentences, manually searching for them across multiple languages is impractical. Existing automated methods, often relying on lexico-syntactic patterns, struggle with the variability of definitional sentence structures (low recall) and can be noisy. This paper addresses the need for a flexible, multilingual, and automated approach to extract definitional information and identify hypernyms from text, overcoming the limitations of prior work which was largely confined to English and required manually curated training data.

Methods

The core of this work is the Word-Class Lattices (WCLs) model, an extension of a supervised lattice-based approach. WCLs model the variability of definitional sentences by learning lattice classifiers from automatically clustered training data.

Key methodological contributions include:

  • Multilingual Training Data Acquisition: A novel, heuristic method for automatically creating training datasets from Wikipedia. This involves:
  • Candidate Definition Extraction: Taking the first paragraph of Wikipedia articles (excluding special pages).
  • POS Tagging and Chunking: Identifying noun, verb, and prepositional phrases.
  • Automatic Annotation: Tagging the defined term (TARGET), definitor (VF), definiens (GF, containing the hypernym), and rest (REST) of the sentence. Hyperlinks immediately following the TARGET are used to identify potential hypernyms.
  • Filtering: Discarding candidates where essential fields are missing.
  • WCL Learning:
    • Star Patterns: Generalizing sentences by replacing frequent words with a wildcard.
  • Sentence Clustering: Grouping sentences based on their star patterns.
  • Lattice Construction: Building separate WCLs for DF, VF, and GF fields for each cluster, marking hypernym words.
  • Classification: Combining DF, VF, and GF lattices to best fit a test sentence, maximizing a confidence score based on coverage and support. Hypernyms are extracted from the GF lattice.

Datasets

  • Training Data:
    • English, French, and Italian Wikipedia dumps: Used to automatically generate training sets. The paper reports the number of Wikipedia pages and the resulting automatically annotated definitions for each language (e.g., ~1.55 million definitions for English).
  • Manually annotated samples: A random sample of 1000 sentences per language was manually annotated as definitional or non-definitional, with correct hypernyms identified for evaluation.
  • Evaluation Data:
    • Wikipedia test sets: The manually annotated samples mentioned above.
  • ukWaC Web corpus: A subset (over 300,000 sentences) used for evaluating performance on an open text corpus, mirroring previous work.

Key Insights and Ideas

  • Wikipedia as a Rich Source for Definition Extraction: The paper demonstrates that Wikipedia, with its structured content and hyperlinks, can be effectively mined to automatically generate large, multilingual training datasets for definition and hypernym extraction. This significantly reduces the manual effort required compared to previous approaches.
  • Robustness Across Languages: The WCL model, when trained on automatically acquired data, shows high performance not only in English but also in French and Italian, highlighting its multilingual generalization capabilities.
  • Impact of Training Data Quality: While the automatic annotation process has some inherent noise (e.g., incorrect hypernym identification), the sheer volume of correctly annotated sentences leads to substantial improvements in performance, particularly in recall.
  • Accessibility through Java API and Web Interface: By releasing a Java API and a web application, the authors make their WCL framework readily usable by researchers and developers, democratizing access to this technology.

Key Results

  • High Performance: The WCLs trained on automatically acquired Wikipedia data achieved high precision and recall. For instance, on the Wikipedia test set, English WCLs achieved 98.5% precision and 78.3% recall. French and Italian also showed strong results.
  • Improved Recall: The automatically acquired training data, covering higher variability, led to improved recall compared to models trained on smaller, manually selected datasets.
  • Comparison with Prior Work: The WCL trained on automatically acquired English data outperformed the WCL trained on the original manually selected dataset from Navigli and Velardi (2010) on the ukWaC corpus (98.9% precision and 57.6% recall vs. 94.8% and 56.5%).
  • Multilingual Success: The paper reports strong definition and hypernym extraction performance across English, French, and Italian.

Limitations and Future Work

  • Annotation Noise: The authors acknowledge that the automatic annotation process can introduce some erroneous annotations. These include:
  • Wikipedia pages not always starting with definitional sentences.
  • Ambiguity in verbal phrases between the defined term and hypernym.
  • First hyperlinks not always accurately identifying the correct hypernym.
  • Future Work: The authors suggest that these annotation errors could be mitigated through language-dependent heuristics or human annotation. They also imply that further improvements could be made by refining the training data acquisition process. The paper also highlights the potential for improving overall performance by manually checking the automatically annotated training datasets.

Summary updated

BibTeX
@inproceedings{faralli2013java,
  author = {Stefano Faralli and Roberto Navigli},
  title = {{A Java Framework for Multilingual Definition and Hypernym Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2013},
}
Figure