EtymoLink: A Structured English Etymology Dataset

Yuan Gao, Weiwei SunACL 2024

TL;DR

This paper introduces EtymoLink, a novel structured dataset of English etymological relationships, automatically extracted from the Online Etymology Dictionary using the FLAN-T5 language model. This work bridges the gap between unstructured lexicographical data and computational linguistics, enabling large-scale analysis of language evolution.

What Problem Does This Paper Address?

Etymology, the study of word origins, is crucial for understanding language evolution and cultural context. However, etymological information is typically buried in unstructured prose within scholarly articles and dictionaries. This format is difficult for computational analysis, hindering large-scale linguistic studies. Existing etymological studies are often specialized and rely on manual, comparative methods. The lack of structured, computationally accessible etymological databases limits the ability of computational linguists to quantitatively analyze historical linguistic changes. This paper aims to address this by creating a structured dataset suitable for computational approaches.

Methods

The authors developed a methodology to automatically extract structured etymological relationships. This involved:

  • Data Crawling: Extracting data from the Online Etymology Dictionary (Etymonline).
  • Data Preprocessing: Cleaning and preparing the text, including handling homographs and preserving hyperlinks for network building.
  • Manual Annotation: A subset of entries (5,361) were manually annotated to create a high-quality ground-truth dataset. This involved converting prose descriptions into an edge list format, representing direct etymological links between word forms and their languages.
  • Automatic Annotation: Fine-tuning the FLAN-T5-base language model on the manually annotated data to automatically extract etymological relationships from the remaining entries. The model was trained to output structured edge lists.
  • Evaluation: The system was evaluated using both string-based metrics (BLEU, ROUGE, ChrF) and edge-based metrics (Edge Recall, Language Label Detection/Accuracy, Word Root Accuracy, Word Root Levenshtein Distance) to assess both textual similarity and structural accuracy.

Datasets

  • Online Etymology Dictionary (Etymonline): The primary source for data extraction.
  • EtymoLink Dataset: A newly introduced dataset containing over 103,000 etymological relationships covering 63,603 English lexical terms. This dataset is derived from Etymonline, with 5,361 entries manually annotated and 58,242 automatically annotated.

Key Insights and Ideas

  • LLMs for Lexicographical Structuring: Large Language Models (LLMs) like FLAN-T5 are highly effective at transforming unstructured, prose-based etymological entries into structured formats suitable for computational analysis. This demonstrates a promising avenue for automating the creation of valuable linguistic resources.
  • The "Plateau Effect" in Training Data: For string-based metrics (like ROUGE and ChrF), performance plateaus after a certain amount of training data (around 3000 examples). This suggests that LLMs can achieve good performance on these metrics with relatively limited manual annotation.
  • Importance of Edge-Based Metrics: While string-based metrics are useful, edge-based metrics are crucial for evaluating the structural accuracy of etymological relationships. The paper highlights that a model can have high string similarity but still fail to capture the correct hierarchical or logical connections.
  • Language Labeling is Challenging: The model showed high accuracy in identifying etymological relationships but struggled with correctly assigning language labels to word roots. This suggests that while LLMs can infer connections, precise linguistic categorization remains a more difficult task.
  • Affixes as Highly Connected Terms: The analysis revealed that affixes (like 'un-', '-y', '-ly') are the most frequently connected lexical terms in the dataset, underscoring their productive role in English word formation.

Key Results

  • The EtymoLink dataset contains over 103,000 etymological relationships.
  • The fine-tuned FLAN-T5 model achieved 94.4% accuracy in correctly identifying relationships between word roots.
  • String-based evaluation metrics showed high accuracy (e.g., BLEU: 0.902, Rouge: 0.920, ChrF: 0.929).
  • Edge-based evaluation metrics demonstrated strong performance, with Edge Recall at 0.905 and Word Root Accuracy at 0.905.
  • Language Label Detection was very high (0.990), but Language Label Accuracy was lower (0.909), indicating a tendency to assign a label but not always the correct one.
  • Removing the language label constraint for evaluation significantly improved the accuracy of relationship identification to 98.2%.

Limitations and Future Work

  • Limitations:
    • The model struggled with accurately assigning language labels to word roots.
  • The dataset is derived from a single source (Etymonline), and the transferability to other dictionaries needs further exploration.
  • While the manual annotation task was designed to be simple, the use of a single annotator might introduce some subjectivity, though the authors argue clarity and simplicity mitigated this.
  • Future Work:
    • Investigating the transferability of the current model to other dictionary datasets, potentially reducing the need for extensive manual annotation on new sources.
  • Further refining the model's ability to accurately identify and assign language labels.
  • Expanding the dataset to include more languages and etymological information.

Summary updated

BibTeX
@inproceedings{gao2024etymolink,
  author = {Yuan Gao and Weiwei Sun},
  title = {{EtymoLink: A Structured English Etymology Dataset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure