TL;DR
This paper introduces a novel dataset of ambiguous noun-adjective multiword expressions (MWEs) in Galician, providing human-annotated compositionality scores at the token level. This resource addresses the critical need for low-resource language data to evaluate how well language models understand nuanced semantic phenomena like idiomaticity and ambiguity.
What Problem Does This Paper Address?
Natural Language Processing (NLP) models often struggle with multiword expressions (MWEs) due to their complex semantic compositionality and the ambiguity of their constituent words. While many MWE datasets exist, they often focus on high-resource languages or provide only type-level annotations. This leaves a significant gap in evaluating models on their understanding of MWEs in low-resource languages like Galician, especially when considering how the same MWE can have different meanings (senses) depending on the context. This paper aims to fill this gap by creating a comprehensive, token-level dataset for Galician MWEs.
Methods
The researchers first extracted noun-adjective compounds from the Galician Wikipedia, ranking them by frequency. They then manually selected 240 compounds, aiming for a diverse range of frequency, compositionality, polysemy, and ambiguity. For each selected MWE, they defined its potential senses (322 in total) and performed a preliminary classification of these senses as compositional, partial, or idiomatic based on constituent transparency.
The core of their contribution is the annotation task. They created two sets of sentences for each sense: two manually written by a language expert and four extracted from corpora. Human annotators (one expert and six external native speakers) were then tasked with rating the compositionality of the MWEs and their individual components (head and modifier) on a scale of 0 (not literal) to 5 (literal), using the provided sentences for context. This allowed for token-level compositionality scores. Additionally, they gathered frequency, ambiguity, and productivity data for the MWEs and their constituents from corpora and lexical resources.
Datasets
The primary dataset introduced in this paper is a collection of 240 Galician noun-adjective MWEs, encompassing 322 distinct senses. This dataset includes:
- Manually created sentences (2 per sense) and corpora-extracted sentences (4 per sense) for contextualization.
- Human judgments on compositionality scores for MWEs, heads, and modifiers at the token level.
- Frequency, ambiguity, and productivity data for MWEs and their constituents.
This dataset is a novel contribution for the Galician language, specifically addressing token-level, context-dependent MWE analysis.
Key Insights and Ideas
- Token-level is Crucial: The paper strongly emphasizes that context matters immensely for MWEs. A single MWE can have multiple senses, and its compositionality can vary significantly, making token-level annotation essential for accurate evaluation.
- Low-Resource Language Challenge: The creation of this dataset highlights the ongoing challenge of building NLP resources for low-resource languages like Galician, where existing annotated data is scarce.
- Human Judgments Reveal Nuance: The human annotation process revealed that the preliminary classification of senses was not always aligned with the final human ratings, suggesting that human perception of compositionality is complex and can differ from initial linguistic intuition. Specifically, a significant portion of senses were rated as less idiomatic by human annotators than initially classified.
- Weak Correlations: Despite gathering various linguistic features, the study found weak and generally non-significant correlations between compositionality scores and frequency, ambiguity, or productivity. This suggests that these factors, while important, don't directly predict compositionality in a straightforward manner, especially in a token-level, context-dependent setting.
Key Results
- The dataset contains 240 noun-adjective MWEs, covering 322 senses, contextualized in 1,858 sentences.
- Human annotators provided compositionality scores for compounds, heads, and modifiers, with inter-annotator agreement scores (Krippendorff's alpha) ranging from 0.584 to 0.705 for compounds, indicating moderate to good agreement.
- Analysis of correlations between compositionality and other linguistic features (frequency, ambiguity, productivity) showed weak and mostly non-significant relationships. For instance, Spearman correlations were generally below 0.2 in magnitude.
Limitations and Future Work
Limitations: * Number of Annotators: The dataset was annotated by a limited number of annotators (one expert and six external annotators). The authors acknowledge that a higher number of ratings per sense would enhance reliability and provide a more robust representation of compositionality degrees. * Galnet Size: The ambiguity data was sourced from Galnet, which is noted as a relatively smaller lexical resource compared to its English counterparts. This limits the depth of ambiguity analysis.
Future Work: * Expand Annotations: Collect more human ratings to further strengthen the dataset's reliability. * Broader MWE Types: Apply the same methodology to create datasets for other types of MWEs, such as verb-object combinations. * Explore Linguistic Properties: Investigate other linguistic properties and contextual cues that might influence human perception of compositionality. * Expand Lexical Resources: Further expand Galnet's synset coverage to allow for a finer representation of constituent ambiguity.
Summary updated