TL;DR
This paper investigates whether treating multiword expressions (MWEs) as single units can improve shallow parsing accuracy. By replacing MWEs with a special token in text and re-parsing, they found a significant accuracy increase of 7.5% to 9.5%, with even higher gains for non-compositional adjective-noun MWEs.
What Problem Does This Paper Address?
Shallow parsing, a fundamental task in Natural Language Processing (NLP) that aims to identify the syntactic structure of sentences (like phrases), often struggles with multiword expressions (MWEs). MWEs are sequences of words that, when taken together, have a meaning that isn't simply the sum of their individual parts (e.g., "kick the bucket" doesn't literally mean to strike a pail with one's foot). Traditional parsers often treat each word in an MWE independently, potentially leading to incorrect syntactic analyses. While the importance of MWEs for parsing has been recognized, there's been a lack of empirical evidence quantifying their impact on shallow parsing accuracy. This paper aims to fill that gap by experimentally measuring the improvement gained by explicitly recognizing and unifying MWEs.
Methods
The core of the methodology involves a clever experimental setup:
- MWE Identification and Replacement: The authors first identified a set of MWEs from WordNet, focusing on two-token sequences like compound nominals, proper names, and adjective-noun constructions. For each identified MWE in a sentence, they replaced it with a single, unique token (e.g., "lemon tree" became "lemon_tree"). This new token was designed to be unknown to the part-of-speech (POS) tagger, forcing it to rely on context rather than its internal lexicon for this token.
- Shallow Parsing Comparison: They then ran a shallow parser on the original sentences and on the modified sentences (with MWEs replaced).
- Change Classification: Crucially, they didn't just compare the outputs. They developed a detailed classification system for the differences observed between the original and modified parse trees. This allowed them to categorize how the parser's output changed (e.g., phrases becoming leaves, leaves becoming phrases, changes in phrase structure). This classification was essential for determining whether a change represented a correction or an error introduced by the MWE replacement.
- Hypothesis for Correctness: To assess if a change was a correction, they used a hypothesis: the "correct" parse tree has the fewest phrases and the fewest unassigned tokens (leaves). This provided a principled way to count instances as improved or degraded.
Datasets
- Multiword Expressions: A set of 118 two-token multiword expressions was randomly selected from WordNet 3.0. These were further categorized as compositional or non-compositional and by the parts of speech of their components.
- Corpora: For each MWE, a corpus of web text snippets containing that MWE was automatically collected. These snippets were then combined to form a large dataset of 376,007 sentences, each potentially containing one or more of the target MWEs.
- Tools: The GENIA tagger was used for part-of-speech tagging, and the SNoW-based Shallow Parser was used for parsing.
Key Insights and Ideas
- MWEs are not just syntactic noise: The study provides strong empirical evidence that treating MWEs as single units significantly benefits shallow parsing, moving beyond anecdotal observations.
- Compositionality matters: Non-compositional MWEs (where the meaning is not predictable from its parts) lead to greater parsing improvements than compositional ones. This highlights the idiomatic nature of these expressions as a key challenge for parsers.
- Adjective-noun MWEs are particularly impactful: These constructions, especially when non-compositional, show the highest parsing accuracy gains, suggesting they are a significant source of parsing errors when treated individually.
- A principled way to evaluate parsing changes: The development of a change classification system, coupled with a hypothesis for identifying correct parses, offers a robust method for evaluating the impact of linguistic features on parsing.
Key Results
- Overall Improvement: Unifying MWE tokens led to an average increase in shallow parsing accuracy of 7.5% to 9.49% on sentences containing these MWEs.
- Adjective-Noun Constructions: These showed higher improvements, ranging from 12% to 15%.
- Non-Compositional Adjective-Noun Constructions: These yielded the highest gains, between 15.5% and 19.5%.
- Compositionality Effect: Compositional MWEs resulted in lower improvements (e.g., 5.54% to 7.19% for noun-noun compositional MWEs) compared to non-compositional ones.
- Most Common Change: The
PL2Pchange class (many phrases/leaves merged into a single phrase after MWE replacement) was the most frequent type of parser output difference, accounting for 34.03% of changes.
Limitations and Future Work
- Focus on Two-Token MWEs: The study is limited to MWEs consisting of exactly two tokens. The authors suggest that longer MWEs might have an even greater impact on parsing accuracy, as their misinterpretation could lead to more significant parsing errors.
- Manual Annotation for Stronger Validation: While their change classification method is innovative, the authors acknowledge that using a treebank with manual MWE annotations would provide a more direct and accurate evaluation, bypassing the need for the change class analysis.
- Parser Integration: The paper suggests that future work should focus on developing parsers that can inherently recognize and integrate MWE knowledge, rather than relying on pre-processing steps like token replacement.
- Generalization: The reported improvements are for sentences containing known MWEs. Projecting this improvement to general text would require estimating the frequency of MWEs in that text.
Summary updated