Predicate Sense Disambiguation for UMR Annotation of Latin: Challenges and Insights

Federica GambaACL 2024

TL;DR

This paper explores using modern language models to automate the tedious task of assigning the correct meaning (sense) to Latin verbs when annotating text for the Uniform Meaning Representation (UMR) framework. The study highlights the significant challenges posed by the limited resources available for Latin, particularly the fine-grained nature of existing lexicons, and evaluates the performance of various pre-trained language models (PLMs) in this context.

What Problem Does This Paper Address?

Annotating text with a structured semantic representation like UMR is crucial for cross-linguistic linguistic analysis. However, a key step in this process is Word Sense Disambiguation (WSD), which involves identifying the correct meaning of a word in its specific context. For languages with scarce resources, like Latin, this task is particularly challenging. Existing resources often simplify word senses into broad categories, which may not be granular enough for detailed semantic annotation. This paper tackles the problem of how to leverage existing, albeit limited, Latin resources and modern NLP techniques (PLMs) to assist in the manual WSD task for UMR annotation, aiming to streamline the laborious annotation process.

Methods

The core of the methodology is a Predicate Sense Disambiguation (PSD) workflow that leverages contextual embeddings from pre-trained language models (PLMs). The approach involves:

  • Extracting Verbal Tokens: Identifying all verbs from the Latin text (Sallust's De Coniuratione Catilinae) using morphological and tectogrammatical annotations from a treebank.
  • Storing Annotated Synsets: For a reference corpus, the manually assigned synsets (word senses) are stored. These are primarily drawn from the Latin WordNet/Vallex resource.
  • Computing and Comparing Embeddings: Contextual embeddings for each verbal token are generated using the Flair library. Cosine similarity is then used to measure the similarity between target tokens and reference tokens.
  • Extracting Candidate Synsets: The top-k most similar tokens from the reference corpus are identified, and their associated synsets are extracted as candidate suggestions.
  • Constraining Candidate Tokens: Initially, the approach focuses on tokens sharing the same lemma (base form) as the target token. This "lemma constraint" was introduced to simplify the task, as preliminary results were not promising. The paper also explores a setting without this constraint to analyze synset retrieval more directly.

The paper evaluates four PLMs: mBERT, LatinBERT, PhilBERTa, and PhilTa.

Datasets

  • Sallust's De Coniuratione Catilinae: The primary text used for annotation and evaluation. It was provided in an XML-based format (Prague Markup Language - PML) and linked to the Latin Dependency Treebank (LDT).
  • Latin Vallex and Latin WordNet (WN): These are the main linguistic resources used for identifying and assigning synsets (word senses) to Latin verbs. They are interlinked within the LiLa Knowledge Base. The paper notes the fine-grained nature and potential limitations of these resources for the task.

Key Insights and Ideas

  • Resource Granularity is a Major Hurdle: The paper strongly emphasizes that the fine-grained nature of existing Latin resources like Vallex/WN, while potentially rich, presents significant challenges for automated WSD. Many verbs have numerous senses with very similar definitions, making it difficult for models to distinguish them.
  • Lemma Constraint Helps, But Isn't Perfect: The introduction of a "lemma constraint" (only considering candidate tokens with the same lemma as the target) improved performance by simplifying the task. However, the authors acknowledge that in real-world scenarios, correct synsets can be found even when lemmas differ.
  • PLM Performance Varies, But No Clear Winner: While PhilBERTa generally performed best in quantitative evaluations, no single PLM consistently outperformed others across all metrics. The performance differences were often subtle.
  • Frequency Matters: The study observed that performance generally improved as the evaluation scope narrowed to more frequent verbs, suggesting that the availability of more training data for common words is beneficial.
  • Manual Annotation Remains Crucial: Even with PLM suggestions, manual post-inspection and annotation are deemed indispensable, especially when synset definitions for retrieved candidates do not align well with the context or are not present in the reference corpus.

Key Results

  • Quantitative Analysis:
    • The paper introduces four metrics: guess_tot, guess_no_hapax, guess_seen, and guess_freq, all measuring the average number of suggested candidates before retrieving one with the same lemma. Lower scores indicate better performance.
  • PhilBERTa generally performed best across these metrics, while PhilTa performed the worst.
  • Performance improved as the evaluation focused on more frequent lemmas.
  • A significant portion of target predicates (20%) were out-of-vocabulary (OOV) in the reference corpus, making lemma-based retrieval impossible.
  • Manual Evaluation:
    • A manual assessment of a subset of results showed that PLMs' performances did not differ substantially when using the lemma constraint.
  • Cases where no correct synset was retrieved were often explained by the sense not appearing in the reference corpus for that specific lemma.
  • Synset retrieval without a lemma constraint still resulted in a high number of attempts before a correct guess, with mBERT outperforming PhilBERTa in one specific instance.

Limitations and Future Work

  • Limited Training Data: A significant limitation is the small size of the annotated dataset, which prevented fine-tuning the PLMs. The authors suggest that fine-tuning would likely enhance performance.
  • Need for Larger Reference Corpus: The study highlights the need for a larger reference corpus to improve results and address data scarcity.
  • Synset Definition Embeddings: An envisioned extension involves computing embeddings for synset definitions themselves. This would allow for comparing definitions directly, potentially handling OOV cases and synsets not present in the reference corpus, and dealing with verbs that share synsets but not lemmas.
  • Sense Clustering: The paper suggests that sense clustering, as described by Navigli (2006) and Martelli et al. (2022), could be a promising avenue to address the granularity challenges of Latin Vallex.

Summary updated

BibTeX
@inproceedings{gamba2024predicate,
  author = {Federica Gamba},
  title = {{Predicate Sense Disambiguation for UMR Annotation of Latin: Challenges and Insights}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure