Extending an Event-type Ontology: Adding Verbs and Classes Using Fine-tuned LLMs Suggestions

Jana Straková, Eva Fučíková, Jan Hajič, Zdeňka UrešováACL 2023

TL;DR

This paper explores using fine-tuned Large Language Models (LLMs) to assist in the manual annotation and extension of a multilingual event-type ontology (SynSemClass). By leveraging LLMs to suggest potential verb-class assignments, the authors aim to make the laborious process of ontology enrichment more efficient and scalable.

What Problem Does This Paper Address?

Building and maintaining comprehensive ontologies, especially for complex domains like event types, is a significant challenge. Manual annotation is expensive, time-consuming, and requires specialized expertise. As ontologies grow and evolve, identifying new concepts (classes) and assigning existing words (verbs) to them becomes increasingly difficult. This paper addresses the need for more efficient methods to extend such ontologies, specifically by using LLMs to pre-annotate data and guide human annotators.

Methods

The core of this work involves fine-tuning a RemBERT (a rebalanced mBERT) model for a classification task. Instead of a traditional multi-class classification, they employ a multi-label approach, treating each class as an independent binary classification problem. This is achieved by using a sigmoid activation function on the output layer and a sparse binary focal cross-entropy loss. This setup is designed to handle the "out-of-distribution" problem where verbs might not fit any existing class, allowing for the identification of potential new classes. The model is trained on a large corpus of Czech sentences annotated within the SynSemClass ontology.

Datasets

The primary dataset used is the Czech part of the SynSemClass 4.0 ontology, which contains example sentences with manually annotated verbs. For training the LLM, a large raw corpus of written Czech (SYN v4) was utilized, specifically the first 2.75 million sentences. The SynSemClass ontology itself is populated with synonymous verbs from English, Czech, German, and Spanish, and links to external resources like Vallex, FrameNet, and VerbNet.

Key Insights and Ideas

  • LLMs as "Election Votes": The paper frames LLM suggestions not as definitive answers but as "election votes" for human annotators, emphasizing that the final decision always rests with the human expert.
  • Two-Pronged Extension Strategy: The approach aims to extend the ontology in two ways: identifying verbs with consistently low class affiliation scores as candidates for new classes, and identifying high-certainty predictions as "low-hanging fruit" for adding verbs to existing classes.
  • Linear Correlation is a Positive Sign: Despite the annotation process not showing significant speedups, the strong, near-linear correlation between LLM scores and human decisions is highlighted as a key positive outcome, indicating the LLM's ability to capture meaningful relationships.
  • The Challenge of "Out-of-Distribution": The paper explicitly tackles the problem of verbs that don't fit existing classes, a common issue in ontology extension, by modeling it as a multi-label problem.

Key Results

  • Strong Correlation: The fine-tuned classifier achieved a strong Pearson's correlation (r = .44) and a moderate Spearman's correlation (ρ = .39) with human annotation decisions.
  • Inter-Annotator Agreement: High inter-annotator agreement (0.83) was observed, though Cohen's kappa was lower (0.51) due to a highly skewed distribution of decisions (mostly rejections).
  • Annotation Time: While the LLM suggestions didn't lead to a significant overall speedup in annotation time, the authors note that the process of annotating thousands of unseen lemmas could be completed within a reasonable timeframe (around 100 hours for double annotation).
  • Thresholds are Context-Dependent: The paper suggests a potential threshold of 0.3 for separating "yes" and "no" decisions, but emphasizes that due to the linear correlation, manual inspection is still necessary across a wide range of scores.

Limitations and Future Work

  • Model Retraining: A significant limitation is that the model needs to be retrained whenever new classes or lemmas are added to the ontology, which is computationally intensive.
  • Skewed Decisions: The annotation process resulted in a high number of rejections, partly due to presenting a fixed number of suggestions regardless of score.
  • Language Specificity: The experiment was conducted primarily on Czech due to its morphological richness and the ontology's coverage. Results for other languages might differ.
  • Future Work: The authors plan to repeat the experiment with larger LLMs and for a larger ontology, exploring the effort required for sustainable development and maintenance. They also suggest investigating methods for more efficient retraining and exploring different language pairs.

Summary updated

BibTeX
@inproceedings{strakova2023extending,
  author = {Jana Straková and Eva Fučíková and Jan Hajič and Zdeňka Urešová},
  title = {{Extending an Event-type Ontology: Adding Verbs and Classes Using Fine-tuned LLMs Suggestions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure