Construction automatique d'une base de données étymologiques à partir du wiktionary

Benoît SagotHAL 2017

TL;DR

This paper presents a method for automatically constructing a large-scale, multilingual etymological database by extracting information from Wiktionary. This work addresses the scarcity of structured etymological data, which is crucial for historical and comparative linguistics, by creating a database of nearly a million lexemes and over half a million etymological relationships.

What Problem Does This Paper Address?

Electronic lexical resources, while abundant, often lack detailed etymological information. This information is vital for understanding language evolution, historical linguistics, and comparative linguistics. Existing etymological databases are often limited in scope, outdated, or lack formal structure, making them difficult to use for computational linguistics tasks. The authors aim to bridge this gap by leveraging the rich, collaboratively-built Wiktionary to create a comprehensive and structured etymological resource.

Methods

The authors developed a multi-stage process to extract and structure etymological data from Wiktionary. This involved:

  • Data Conversion: Converting the Wiktionary dump into a structured XML format.
  • Etymological Section Extraction: Identifying and extracting sections dedicated to etymology within Wiktionary articles.
  • Form Normalization: Standardizing the representation of word forms (e.g., handling affixes, compounds) using XML tags and symbols like '+' and '~'.
  • Glose Inference: Developing patterns to infer missing "glosses" (definitions) for word forms by leveraging context from the main entry's definition.
  • Relation Extraction: Using regular expression patterns to identify and extract etymological relationships (e.g., inheritance, borrowing, derivation) between lexemes.
  • Lexeme Fusion: Implementing an iterative algorithm to merge synonymous lexemes that might have different representations or glosses in Wiktionary, aiming for a unified representation.
  • Direct Relation Enforcement: Removing indirect relations to ensure that extracted relationships are direct links between lexemes.
  • Relation Type Correction: Refining the classification of relations to more accurately distinguish between inheritance, borrowing, and morphological derivation.
  • TEI Export: Developing a module to export the constructed etymological relations into the TEI (Text Encoding Initiative) format, extending existing proposals to accommodate cognation and alternative hypotheses.

Datasets

The primary dataset used in this paper is the English edition of Wiktionary, specifically a dump from January 1, 2017. The authors also compare their results with EtymWordNet, a previously constructed etymological database derived from Wiktionary.

Key Insights and Ideas

  • Wiktionary as a Rich Source: The paper demonstrates that Wiktionary, despite its collaborative and sometimes informal nature, contains a significant amount of valuable etymological information that can be systematically extracted.
  • The Importance of Lexemes over Lemmas: The authors highlight that for etymological analysis, working with "lexemes" (specific word forms with their language and meaning) is more appropriate than "lemmas" (dictionary headwords), as it allows for a more granular and accurate representation of historical language changes.
  • Challenges in Data Extraction: The semi-structured nature of Wiktionary, the variety of templates used, and the diverse ways contributors represent information pose significant challenges for automated extraction.
  • The Need for Relation Typing: Distinguishing between different types of etymological relationships (inheritance, borrowing, derivation) is crucial for accurate linguistic analysis, a feature that was lacking in previous comparable datasets like EtymWordNet.
  • Building a Foundation for Computational Linguistics: The created database serves as a foundational resource for computational historical linguistics, enabling tasks like automated phonetic law verification and the construction of comprehensive etymological dictionaries.

Key Results

  • The process resulted in a multilingual database of nearly one million lexemes and a database of more than half a million etymological relationships between lexemes.
  • The initial extraction produced approximately 1.2 million lexemes, 62,056 sequences of lexemes, and 548,935 etymological relations. After fusion and filtering, the final database contains 975,473 distinct lexemes and 514,199 distinct relations.
  • The lexemes in the database span 2311 distinct languages, with English, Latin, French, Italian, and Ancient Greek being the most represented.
  • 68% of the lexemes have a gloss (definition).
  • A manual evaluation of 100 relations indicated that 78 were correct, with 18 misclassified as "heritage" when they were actually "borrowings."
  • Comparison with EtymWordNet showed that the authors' database is significantly richer, connecting more lexemes, including compounds and derivatives explicitly, and providing more precise relation types.

Limitations and Future Work

  • Glose Inference Accuracy: While the glose inference algorithm is effective, there are still occasional errors due to unusual wiki syntax or misinterpretations of definitions.
  • Lexeme Fusion Completeness: The current lexeme fusion algorithm is not exhaustive and could be improved to unify more synonymous lexemes.
  • Relation Typing for Borrowings: A significant portion of relations were misclassified as "heritage" when they were actually "borrowings," highlighting the need for more explicit language-pair information.
  • Handling Alternative Hypotheses: The current TEI export format does not fully accommodate alternative etymological hypotheses, which are common in the data.

Future work suggested includes:

  • Extending Pattern Matching: Refining and expanding the patterns used for structuring etymological notices and extracting lexemes/relations.
  • Improving Lexeme Fusion: Enhancing the fusion algorithm to handle more variations in form and gloss.
  • Leveraging External Resources: Using external lexical resources like WordNet or distributional similarity bases to improve glose attribution.
  • Phylogenetic Language Modeling: Incorporating language phylogenetic trees to better model indirect relations and reconstruct language evolution.
  • Extracting from Other Wiktionary Editions: Expanding the extraction to include other language editions of Wiktionary, such as the French Wiktionnaire.
  • Automated Phonetic Law Verification: Using the database to automatically verify phonetic laws and their systematic applicability.
  • Building Comprehensive Etymological Dictionaries: Ultimately aiming to construct or complete large-scale etymological dictionaries.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{sagot2017construction,
  author = {Benoît Sagot},
  title = {{Construction automatique d'une base de données étymologiques à partir du wiktionary}},
  booktitle = {HAL},
  year = {2017},
}
Figure