TL;DR
This paper systematically compares twelve statistical association measures for automatically identifying collocations in English, Portuguese, and Spanish, using a novel, manually annotated gold-standard corpus. The study reveals that simple frequency counts often perform as well as more complex measures, and that symmetric measures generally outperform directional ones, even though collocations are inherently asymmetric.
What Problem Does This Paper Address?
Identifying collocations – words that frequently co-occur in a predictable way (e.g., "strong coffee," not "powerful coffee") – is crucial for many natural language processing tasks like machine translation and understanding. While statistical association measures (AMs) are commonly used for this, there's a lack of systematic comparison across different languages, syntactic patterns, and AM types. Furthermore, the inherent asymmetry of collocations (one word is "base," the other "collocate," with restricted choice) suggests directional measures might be superior, but this hasn't been thoroughly tested. This paper aims to fill this gap by providing a comprehensive evaluation of various AMs on a multilingual, syntactically annotated dataset.
Methods
The authors employed a phraseological approach to identify collocations, focusing on dependency triples derived from syntactic parsing. They evaluated twelve statistical association measures, including both symmetric (e.g., log-likelihood, t-score, mutual information) and directional (e.g., DeltaP, NPMI) measures. The evaluation involved computing precision and recall for each measure across different collocation types (adjective-noun, verb-object, nominal compounds) and languages. Performance was visualized using precision-recall curves and summarized using Area Under the Curve (AUC) and Mean Average Precision (MAP).
Datasets
The study utilized a newly created gold-standard corpus for English, Portuguese, and Spanish. This corpus was derived from the PARSEME 1.1 Shared Task corpora, with manual annotation of 1,526 collocations across three syntactic patterns (adjective-noun, verb-object, nominal compounds) based on the Meaning-Text Theory. For training the statistical models to compute association values, three reference corpora were compiled, each containing approximately 100 million tokens from sources like Wikipedia, Europarl, and OpenSubtitles.
Key Insights and Ideas
- Frequency is King (or close to it): Simple raw frequency often performs as well as, or even better than, many sophisticated statistical association measures for identifying collocations, especially in certain syntactic patterns. This challenges the assumption that complex measures are always necessary.
- Symmetry Trumps Directionality: Despite the theoretical asymmetry of collocations, symmetric association measures generally outperformed directional ones. This suggests that capturing the "base-collocate" relationship might be more effectively achieved through symmetric measures in practice.
- Syntactic Patterns Matter: The performance of association measures varies significantly depending on the syntactic pattern of the collocation. Verb-object collocations were the easiest to extract, while nominal compounds proved the most challenging.
- Language Independence: The relative performance of most association measures was remarkably similar across the three evaluated languages, suggesting that general trends can be observed.
Key Results
- Raw frequency was identified as one of the best-performing measures across languages and patterns, often achieving the highest AUC and MAP scores.
- Directional measures like DeltaP and NPMI generally showed lower performance compared to symmetric measures.
- Verb-object collocations had the highest extraction accuracy, while nominal compounds had the lowest.
- The study found that most collocations in their dataset occurred with fewer than 3,000 occurrences in the reference corpora.
Limitations and Future Work
The authors acknowledge that even the best-performing measures do not achieve perfect precision, indicating that single statistical measures are insufficient for fully automating collocation extraction. They suggest that future work could focus on:
- Combining AMs: Exploring combinations of different statistical measures to improve performance.
- Distributional Approaches: Utilizing distributional semantics to classify MWEs based on their compositionality, which could help filter out non-collocational expressions.
- Pattern-Specific AMs: Developing or applying specific AMs tailored to different syntactic patterns and frequency ranges (e.g., for low-frequency collocations).
- Semantic Compositionality: Investigating semantic compositionality to better identify idioms and other non-collocational candidates.
Summary updated