TL;DR
This paper pioneers the study of how the "transparency" of noun-noun compound meanings changes over time. By leveraging the Google Books Ngram corpus and adapting methods from semantic change detection, they demonstrate that temporal information can improve predictions of compositionality, offering a new lens on linguistic evolution.
What Problem Does This Paper Address?
Compositionality, the idea that an expression's meaning is derived from its parts and their structure, is a cornerstone of linguistics. While researchers have developed computational methods to measure how "transparent" a compound's meaning is (e.g., how predictable "hot dog" is from "hot" and "dog"), this has largely been a synchronic endeavor. The authors identify a significant gap: the diachronic aspect of compositionality – how this transparency evolves over historical periods – has been virtually unexplored. They aim to bridge this gap by investigating whether and how the compositionality of noun-noun compounds changes over time.
Methods
The core of the methodology involves two main components:
- Semantic Space Construction: The authors utilize the Google Books Ngram corpus, which provides time-stamped data (from the 1500s to the 2000s). They extract 5-grams and use the words within them as both target words and context. To capture the meaning of compounds and their constituents, they build semantic spaces for each year. A "compound-centric" setting is adopted, where word representations are based on their usage specifically within compounds, inspired by previous work suggesting this outperforms general word embeddings for predicting novel compounds.
- Compositionality Measurement: They adapt existing methods for predicting compositionality. Six features are employed: three based on cosine similarity between word embeddings (similarity between constituents, similarity of compound with head, similarity of compound with modifier) and three based on information theory (Log Likelihood Ratio - LLR, Positive Pointwise Mutual Information - PPMI, and Local Mutual Information - LMI). These features are used to predict human-annotated compositionality ratings.
- Diachronic Analysis: To study changes over time, they adapt techniques from semantic change detection. This involves analyzing how the calculated compositionality scores for compounds evolve across different time spans (e.g., 20-year intervals). They jointly learn dense representations using SVD and apply row normalization to mitigate frequency biases.
Datasets
- Google Books Ngram Corpus: Used for building time-stamped semantic spaces and extracting contextual information for compounds and their constituents. This is the primary data source for the diachronic analysis.
- REDDY Dataset: A benchmark dataset compiled by Reddy et al. (2011) containing human ratings for the compositionality of 90 English noun-noun compounds. This dataset is used for evaluating the synchronic performance of their methods and validating the quality of the Google Books Ngram data for this task.
Key Insights and Ideas
- Temporal Information Matters: The study demonstrates that incorporating temporal information into semantic models can indeed improve the prediction of compositionality, suggesting that compound transparency is not static.
- Information Theory Features Shine: Features derived from information theory (LLR and LMI) consistently outperform cosine similarity-based features in predicting human compositionality ratings, even in synchronic evaluations. This suggests that measures of statistical association are more robust for capturing compositionality than simple semantic similarity.
- Modifier vs. Head Influence: Contrary to some prior expectations, the study found that modifiers do not necessarily have a stronger influence on compositionality than heads, and that the relationship is complex and captured well by their chosen metrics.
- Diachronic Separation: When analyzing compounds grouped by their initial compositionality ratings (low, medium, high), the study observed a noticeable separation between these groups in later years, particularly between the 1940s and 1960s. This indicates that compounds with different initial transparency levels can diverge in their temporal evolution.
- Less Compositional Compounds Evolve More: The authors hypothesize that less compositional compounds might undergo more pronounced changes in their transparency over time, a trend that seems to be supported by their findings.
Key Results
- The best synchronic correlation (Spearman's ρ) achieved for predicting human compositionality ratings (compound-mean) was 0.54 using LMI.
- Information theory-based features (LLR, LMI) were found to be the best predictors for compositionality ratings.
- Using temporal information (e.g., 20-year time spans) generally outperformed non-temporal setups in predicting compositionality.
- The best configuration for prediction involved a 20-year time span and a frequency cut-off of 100 for compounds and their constituents.
- The study observed a clear separation in the temporal progression of compositionality for compounds initially rated as low, medium, and high in transparency.
Limitations and Future Work
- Data Limitations: The authors acknowledge that the Google Books Ngram corpus, while vast, has limitations. The context window for constructing word vectors is smaller (5-grams) compared to some previous studies, which might affect performance. They also note that the data for earlier decades is sparser, requiring careful interpretation.
- Synchronic vs. Diachronic Correlation: While temporal information improves predictions, the overall correlation with synchronic ratings from the REDDY dataset is lower than reported in some previous studies. This is attributed to potential differences in datasets, context window sizes, and the compound-centric approach.
- Future Work:
- Expand the models to other languages (e.g., German) for which compositionality ratings are available.
- Further investigate why information theory-based measures outperform cosine similarity-based ones.
- Incorporate more compounds and their ratings, and utilize larger corpora.
- Explore alternative ways to "harvest" proxies for compositionality over time, such as using dictionary information.
Summary updated