TL;DR
This paper tackles the challenge of accurately labeling complex predicates (like "take care") in Natural Language Processing. They propose an automatic method to "alias" these multi-word expressions to existing single-word predicate rolesets in PropBank, significantly reducing manual annotation effort and improving semantic understanding. Their approach achieves competitive accuracy, approaching human annotation levels.
What Problem Does This Paper Address?
Semantic Role Labeling (SRL) aims to understand "who did what to whom" in a sentence by identifying predicates and their arguments. PropBank (PB) is a widely used corpus for training SRL systems, but it primarily focuses on single-word predicates. Multi-word expressions (MWEs), particularly complex predicates (CPs) like light verb constructions (e.g., "take care") and verb particle constructions (e.g., "watch out"), are often treated as simple lexical verbs. This leads to a loss of semantic information because the unified meaning of the CP is missed, and instead, the meaning of its individual components is captured. For instance, "take care" might be annotated with the roleset for "take" (meaning acquire), losing the specific meaning of "being concerned." This is a significant issue as CPs are frequent in language.
Methods
The authors propose an automatic method to find suitable PropBank rolesets (aliases) for complex predicates. Their approach builds on the concept of "aliasing," where CPs are mapped to existing rolesets that capture their meaning and argument structure, thus avoiding the creation of new rolesets.
The core of their method involves:
- Extracting CPs and PB Roleset Groups: They identify CPs from annotated corpora and group single predicates by their assigned PropBank rolesets.
- Multilingual Distributional Model: They leverage a multilingual parallel corpus and word alignments. The hypothesis is that expressions sharing many translations are likely synonymous. They find translations of CPs and single predicates.
- Co-occurrence Matrix and Similarity: They build a co-occurrence matrix where CPs and PB roleset groups are the "head terms." Translations serve as features. They then calculate the similarity between CP vectors and roleset group vectors using cosine similarity.
- Alias Selection: The PB roleset group with the highest similarity score to a CP is selected as its alias.
This method aims to automate the manual aliasing process, making it scalable.
Datasets
- Wiki50 corpus: Used for identifying and extracting complex predicates (LVCs and VPCs). This corpus provides MWE annotations.
- Europarl corpus: A large parallel corpus (approx. 2 million sentences in English) used for training the distributional model. Annotations from MATE tools (lemmas, POS, dependency structures, SRL) were extracted from the English portion.
- PropBank (PB): The target resource for finding alias rolesets.
- Bonial et al. (2014) dataset: A subset of CPs involving the verb "take" was used for evaluation and comparison.
A new, frequency-balanced, data-driven evaluation set was constructed from the Wiki50 corpus, comprising 197 annotated instances of 100 CP types. This set was designed to be more diverse and representative than previous benchmarks.
Key Insights and Ideas
- Automating Aliasing is Feasible: The paper demonstrates that finding semantic role aliases for complex predicates can be automated, significantly reducing the manual effort required for PropBank expansion.
- Multilingualism as a Strength: Using multilingual parallel corpora and word alignments proves effective for capturing semantic similarities between complex predicates and existing rolesets.
- Loss of Information in Current PB: The current PropBank annotation of CPs as individual verbs leads to a significant loss of semantic nuance, which their aliasing approach aims to recover.
- Frequency Matters: The performance of their automatic method is better on higher-frequency CPs, which is expected for distributional methods and highlights the importance of a balanced evaluation set.
Key Results
- Accuracy: The system achieved 44.1% strict accuracy on the entire Wiki50 evaluation set and 69.0% lenient accuracy. For more frequent test items, lenient accuracy reached 81.0%.
- Coverage: The system achieved high coverage, suggesting at least one alias for over 98% of the CPs in the Wiki50 set.
- Comparison to Baseline: Their system generally outperformed a strong baseline (which distinguished VPCs and LVCs and used existing PB entries) in terms of harmonic mean of coverage and accuracy, especially for the "alias" subset (CPs for which a true alias was provided).
- Human Annotation Agreement: The inter-annotator agreement for the human annotation effort was around 53% (micro-averaged IAA), indicating the inherent difficulty and subjectivity of the task, and placing the system's performance in context.
Limitations and Future Work
- Domain Specificity of Parallel Corpus: The use of the Europarl corpus, primarily focused on parliamentary proceedings, might limit the generalizability of the learned features to other domains. Adding parallel data from different domains is suggested.
- Reliance on SRL Output: The method's performance is dependent on the quality of the SRL system used to initially assign rolesets. Errors in the SRL output can be propagated.
- Word Sense Disambiguation: The system tends to assign the most frequent sense of a CP, lacking explicit word sense disambiguation.
- MWE Translation Variability: Complex predicates that maintain their multi-word structure across languages can have less overlap in translations with simplex PB rolesets, posing a challenge. Applying frequency weights to the roleset matrix is proposed.
Future work includes improving the method by addressing these error sources, incorporating more sophisticated techniques, and evaluating the impact of the new CP representation on downstream SRL applications by retraining SRL systems.
Summary updated