TL;DR
This paper explores how the historical evolution of word meanings (diachrony) can help predict the current semantic transparency (compositionality) of English noun compounds. By analyzing word usage patterns over time using both co-occurrence and topic models, the researchers found that historical context significantly improves predictions, with modifier-compound relationships being particularly informative.
What Problem Does This Paper Address?
Understanding how the meaning of a compound word is derived from its constituent parts (compositionality) is a fundamental challenge in Natural Language Understanding (NLU). While many approaches focus on the current state of language (synchronic), this paper investigates whether looking at how word meanings have changed over time (diachronic) can offer a better understanding of present-day compositionality. For example, "climate change" is compositional (meaning is clear from "climate" and "change"), but "snake oil" is not (meaning "false panacea" cannot be derived from "snake" and "oil"). The authors aim to leverage historical linguistic data to build better computational models for predicting this semantic transparency.
Methods
The core of this research involves creating diachronic representations of noun compounds and their constituents using two main approaches:
- Co-occurrence Space: This method uses word embeddings derived from how words appear together in text over time. They analyze the cosine similarity between vectors representing compounds, their modifiers, and their heads across different historical periods.
- Topic Modeling Space: This approach uses stochastic block models (SBMs) to identify latent topics within the text data. Similar to the co-occurrence space, they then calculate cosine similarities between compound and constituent topic distributions over time.
These diachronic similarity scores are then used as features in a binary classification task to distinguish between compounds with low and high present-day compositionality. They also compare these diachronic methods to a static approach that only uses the most recent data.
Datasets
The primary dataset used is the clean version of the Corpus of Historical American English (CCOHA). This corpus contains texts spanning from the 1810s to the 2000s, providing a rich source of diachronic linguistic data. For evaluation, they use a gold standard dataset of English noun compounds with human-annotated compositionality ratings (on a scale of 0-5), which includes compounds from previous work (Reddy et al., 2011; Cordeiro et al., 2019). They focus on a subset of 210 compounds where both constituents are nouns.
Key Insights and Ideas
- History Matters: Diachronic information (how word meanings evolve) is surprisingly effective at predicting present-day compositionality, often outperforming static models that only look at current language use.
- Modifiers Lead the Way: The relationship between a compound and its modifier (e.g., "climate" in "climate change") is a stronger predictor of compositionality than the relationship between the compound and its head, or the relationships between both constituents.
- Topic Models are Competitive: Despite having fewer dimensions than co-occurrence models, topic models perform comparably well in capturing diachronic semantic information relevant to compositionality. This suggests that topic-based representations can be efficient and effective.
- Prepositional Paraphrases Add Value: Including prepositional variants of compounds (e.g., "change in climate" for "climate change") as features helps improve prediction accuracy, indicating their role in understanding semantic relationships.
- Misclassifications Reveal Trends: When the model makes errors, particularly for highly compositional compounds, their historical development often mirrors the patterns of less compositional compounds, suggesting a convergence or blurring of semantic boundaries over time.
Key Results
The study achieved accuracy scores above 0.7 in predicting present-day compositionality using diachronic features. The best results were obtained using compound-modifier similarity, reaching accuracies of 0.745 for topic models and 0.741 for co-occurrence models (fine-grained time slices). This significantly outperforms a random baseline of 0.5. The static approach (using only the last time slice) performed better in co-occurrence experiments, but the diachronic approach was more effective with topic models.
Limitations and Future Work
- Static vs. Diachronic Trade-off: While diachronic models show promise, the static approach (using the most recent data) still performs better in some co-occurrence scenarios, suggesting that for some aspects, recent usage is the most predictive.
- Topic Model Dimension Choice: The performance of topic models is sensitive to the number of topics chosen, with higher numbers (like 2164) yielding better results. Further research could explore optimal topic model configurations.
- Qualitative Analysis of Errors: The paper notes that misclassified high-compositional compounds sometimes exhibit diachronic patterns similar to low-compositional ones. Future work could delve deeper into why these specific items are misclassified and what linguistic phenomena are at play.
- Broader Linguistic Phenomena: The authors suggest that their findings could be extended to other types of multi-word expressions and that diachronic analysis could be applied to other semantic phenomena beyond compositionality.
Summary updated