Abstract
This study investigates the feasibility of automatically identifying Multiword Units (MWUs)—phrases like compact disk or Iwo Jima—for dictionary inclusion using unsupervised, "knowledge-free" methods. The authors evaluate nine standard statistical collocation-finding algorithms and explore the integration of Latent Semantic Analysis (LSA) to model semantic properties like non-compositionality and non-substitutability. Results indicate that while information-theoretic algorithms significantly outperform frequency-based methods, the best models achieve a Figure of Merit of only 0.282. The marginal 1.3% gain from LSA suggests that fully automated, knowledge-free MWU induction remains an open challenge.
Introduction
Multiword Units (MWUs) are sequences of words that function as a single semantic unit. Lexicographers typically include them in dictionaries if they exhibit non-compositionality (the meaning is not a simple sum of parts) or non-substitutability (components cannot be replaced by synonyms without losing the specific meaning). Automating the identification of these units in large text corpora is a long-standing goal in Natural Language Processing. This paper benchmarks existing statistical tools to determine if they can truly identify "dictionary-worthy" headwords without human intervention.
Methodology and Benchmarking
The authors tested nine probabilistic algorithms on a 6.7 million-word subset of the TREC database. These algorithms were categorized into "information-like" methods (e.g., Z-score, $\chi^2$, Pointwise Mutual Information) which detect statistical surprises, and "frequency-like" methods (e.g., T-score, raw frequency) which favor common sequences.
As shown in the rankings above, information-like algorithms (left columns) tend to identify more specific, "dictionary-like" entities such as Iwo Jima or Buenos Aires, whereas frequency-based methods (right columns) often capture common but less lexically distinct phrases like United States or Los Angeles.
Semantic Enhancement via LSA
To improve performance, the researchers applied Latent Semantic Analysis (LSA) to capture semantic nuances. They hypothesized that MWUs should be semantically distinct from the sum of their parts (non-compositionality) and should belong to "substitution sets" (e.g., bachelor's degree and master's degree).
Interestingly, using LSA to detect non-compositionality actually decreased performance. This "compositionality paradox" occurs because, in a corpus, the components of an MWU (like Buenos and Aires) appear almost exclusively together, causing the model to treat them as semantically identical to the full phrase. However, using LSA to identify non-substitutable units—phrases that share a structure with other known MWUs—yielded a modest performance boost.
Results and Discussion
The performance was measured using a "Figure of Merit" (FOM), defined as the area under the precision-recall curve. The best baseline statistical models achieved an FOM of approximately 0.265 to 0.269 when compared against gold standards like WordNet and internet dictionaries.
The integration of LSA-based rescoring improved the best score to 0.282, an absolute gain of 1.3%. While this improvement is statistically visible on the precision-recall curve, it comes at a high computational cost. The large gap between the achieved scores and a perfect 1.0 indicates that current unsupervised methods struggle to distinguish between legitimate dictionary headwords and phrases that are merely frequent within a specific domain.
Conclusion
The study concludes that knowledge-free induction of MWU dictionary headwords is far from a solved problem. While statistical methods provide a useful starting point, they remain noisy. Future work may require more sophisticated models of semantic non-compositionality or "weakly-supervised" approaches that incorporate minimal human knowledge to better align with lexicographic standards.
Summary updated