TL;DR
This paper provides a comprehensive evaluation of different techniques for automatically identifying multiword expressions (MWEs) in text. By comparing four distinct approaches across various languages, MWE types, and corpus sizes, the authors offer valuable insights into their strengths, weaknesses, and practical considerations like computational cost and flexibility, guiding researchers towards the most suitable methods for specific scenarios.
What Problem Does This Paper Address?
Multiword expressions (MWEs) – like "raining cats and dogs" or "take off" – are crucial for natural language understanding and generation. However, automatically identifying these expressions from text is challenging due to their heterogeneity and flexibility. Existing methods have often been evaluated in isolation, on different datasets or languages, making it difficult to compare their effectiveness and determine which approach offers the best "cost-benefit ratio." This paper aims to bridge this gap by conducting a broad, comparative evaluation of several MWE acquisition techniques, considering practical aspects beyond just accuracy.
Methods
The authors evaluate four distinct MWE acquisition approaches:
- LocalMaxs: This method identifies MWE candidates by finding local maxima in association measure distributions. It's language-independent but doesn't directly integrate linguistic information. They tested both strict (high precision) and relaxed (high recall) versions.
- MWE toolkit (mwetk): This is an integrated framework that combines linguistic (POS, lemmas) and frequency information for MWE acquisition. It allows for user-defined criteria and provides tools for preprocessing and evaluation.
- Ngram Statistics Package (NSP): A traditional approach for statistical analysis of n-grams, NSP offers tools for counting and calculating various association measures. While it doesn't directly support linguistic information, it can be simulated.
- UCS toolkit: This toolkit focuses on sophisticated association measures for bigrams. It requires a pre-existing list of candidates and their frequencies, relying on external tools for preprocessing.
The evaluation separates candidate extraction from the filtering/ranking step using association measures (AMs).
Datasets
The experiments were conducted using fragments of the Europarl corpus for both English (en) and French (fr) across three sizes: Small (5,000 sentences), Medium (50,000 sentences), and Large (500,000 sentences).
Gold standards used for evaluation included:
- English: WordNet 3, Cambridge Dictionary of Phrasal Verbs, VPC (Verb-Particle Constructions) dataset, and CN (Noun Compound) dataset.
- French: Lexique-Grammaire.
Key Insights and Ideas
- Candidate lists are surprisingly similar: Despite different extraction strategies, the initial lists of MWE candidates generated by most approaches are highly overlapping, suggesting that the core challenge lies in effectively filtering and ranking these candidates.
- Trade-offs between precision and recall: For nominal MWEs, approaches generally achieve high recall but lower precision. For verbal MWEs, LocMax-S shows high precision but low recall, while others offer a more balanced but lower performance.
- Association Measures matter, but "glue" is king for verbs: The "glue" association measure, specifically used with LocalMaxs, demonstrated exceptionally high mean average precision (MAP) for verbal MWEs, significantly outperforming other measures. For nominal MWEs, several measures showed comparable performance.
- Computational cost is a significant factor: UCS is the slowest approach, while NSP and LocMax-S are the fastest. Memory limitations can also be an issue, with NSP requiring substantial memory for larger n-grams.
- Flexibility varies: While UCS is limited to bigrams, mwetk and LocMax can handle variable n-gram lengths. mwetk stands out for its integration of linguistic filters, though simulating them with other tools yielded comparable results.
Key Results
- Candidate Extraction: For nominal MWEs, approaches achieved around 80% recall and 70% precision. For verbal MWEs, LocMax-S achieved ~70% precision but low recall, while other methods had more balanced but lower P/R values (~20%).
- Association Measures: The "glue" AM with LocMax achieved the highest MAP (87.06%) for verbal expressions. For nominal expressions, several AMs (e.g., Poisson.pv, local.MI) performed comparably to the best (around 60-70% MAP).
- Computational Time: For nominal expressions, time increased linearly with corpus size. For verbal expressions, it increased faster. UCS was the slowest, while NSP and LocMax-S were the fastest.
- Candidate Overlap: LocMax candidates were largely covered by those extracted by other approaches, indicating that LocMax might miss some MWEs that are captured by broader extraction methods.
Limitations and Future Work
The authors acknowledge that their evaluation of candidate extraction provides an underestimation of true MWEs, as candidates not in the gold standard might simply be absent from it. They also note that the evaluation of French verbal expressions was limited due to a lack of resources.
Future work suggested includes:
- Extending the evaluation to consider the domain and genre of source corpora.
- Evaluating other MWE-related tasks like classification and interpretation.
- Investigating dictionary-based identification and bilingual MWE acquisition.
- Conducting application-based extrinsic evaluations with manual validation to demonstrate the practical usefulness of MWE acquisition techniques.
- Exploring the combination of approaches using machine learning.
Summary updated