TL;DR
This paper introduces the first evaluation dataset for detecting semantic change in Modern Chinese, covering a 50-year period around China's pivotal "Reform and Opening-up" policy. By adapting the DURel framework and collecting human judgments, they provide a crucial benchmark for computational models, demonstrating that static word embeddings can achieve significant correlation with human assessments of semantic shift.
What Problem Does This Paper Address?
Understanding how word meanings evolve over time, known as diachronic semantic shift, is a fundamental challenge in linguistics and natural language processing. While computational methods for detecting this shift have advanced, particularly using word embeddings, a significant bottleneck has been the lack of evaluation datasets for many languages. Existing datasets are often limited to binary "changed" vs. "unchanged" labels, failing to capture the nuanced, cumulative nature of meaning evolution. Chinese, a major global language, has been particularly underserved, with no dedicated evaluation datasets for lexical semantic change. This paper aims to fill this gap by creating a dataset for Modern Chinese and evaluating existing computational approaches on it.
Methods
The authors adapted the Diachronic Usage Relatedness (DURel) framework, originally developed for German, to create their Chinese dataset. This framework involves human annotators rating the semantic relatedness of target words across different time periods.
- Corpus Selection: They utilized the People's Daily newspaper corpus, a large, publicly accessible collection of Chinese texts spanning from the 1950s to the early 2000s.
- Time Period Division: The corpus was split into two sub-corpora: C1 (1953-1978) and C2 (1979-2003), demarcated by the "Reform and Opening-up" policy, a significant socio-economic event believed to have influenced language.
- Word List Creation: A list of 20 words was curated: 10 identified as having undergone semantic change and 10 as stable counterparts. These were selected based on frequency, linguistic references, and manual inspection of sampled sentences.
- Annotation: Five native Mandarin Chinese speakers with linguistics backgrounds were recruited. They were asked to rate the semantic relatedness of target words in usage pairs (sentences from C1 and C2) on a four-point scale (Unrelated to Identical), following the DURel methodology.
- Evaluation Metrics: Two metrics derived from the DURel framework were used:
- ∆LATER: Compares average relatedness scores between context pairs within C1 and within C2. It's sensitive to words acquiring new senses.
- COMPARE: Directly compares relatedness scores of usage pairs across C1 and C2. It's considered more robust for words whose semantic shift is complete.
- Model Evaluation: Static word embedding models (Skip-gram and CBOW) were trained on each sub-corpus. These embeddings were then aligned into a shared space to compare word representations across time. The performance of these models was evaluated against the human-annotated dataset using Spearman correlation with the ∆LATER and COMPARE metrics.
Datasets
- People's Daily Corpus: The primary source for constructing the evaluation dataset. Texts range from the 1950s to the early 2000s.
- ZhShiftEval (New Dataset): Introduced in this paper, this is the first human-annotated evaluation dataset for diachronic semantic change in Modern Chinese. It comprises 6,000 human judgments for 20 target words across two time periods (1953-1978 and 1979-2003).
- COS960: A Chinese word similarity dataset used for preliminary evaluation of the trained word embedding models.
Key Insights and Ideas
- The "Reform and Opening-up" as a Linguistic Catalyst: The paper posits that this major socio-economic shift likely drove significant lexical changes in Modern Chinese, justifying the chosen time split.
- ∆LATER vs. COMPARE Nuances: The study highlights how ∆LATER is better at detecting words acquiring new senses, while COMPARE is more effective for words that have already shifted or for capturing shifts in polysemous words where multiple senses compete.
- Homographs and Semantic Competition: The analysis of words like '机制' (mechanism) and '拐' (crutch/traffic) suggests that detected "shifts" might sometimes reflect the competition between different meanings of the same surface form (homographs) rather than a complete loss or gain of senses.
- Static Embeddings Still Relevant: Despite the rise of contextualized embeddings, the paper's evaluation shows that traditional static word embeddings, particularly Skip-gram, can achieve surprisingly strong performance in detecting semantic change when properly aligned.
Key Results
- Dataset Creation: The ZhShiftEval dataset was successfully created with 6,000 human judgments, yielding an inter-annotator agreement (Krippendorff's alpha) of 0.515, comparable to similar datasets in other languages.
- Model Performance:
- Skip-gram embeddings achieved a Spearman correlation of 0.584 with the COMPARE metric and -0.625 with the ∆LATER metric (absolute value), both statistically significant.
- The performance was positively correlated with COMPARE and negatively with ∆LATER, aligning with expectations.
- Qualitative Findings: The study identified words like '推出' (push out/launch) and '机制' (mechanism) as showing strong semantic change, consistent with linguistic literature. It also identified '拐' (crutch/traffic) as a changed word, despite being a control, due to a shift in its dominant usage.
Limitations and Future Work
- Limited Word Set: The current dataset is based on only 20 target words, which is a small sample size for comprehensive evaluation.
- Corpus Bias: The reliance on the People's Daily corpus might introduce biases, as certain topics or word usages might be more prevalent in a newspaper context.
- Future Directions:
- Scaling Up: The authors aim to expand the dataset to include more words and potentially finer-grained time intervals.
- More Models: Evaluating more advanced models, including contextualized embeddings (like BERT), is a key next step.
- Diachronic Variation: Exploring diachronic variation across different Chinese dialects is also proposed as a future research avenue.
Summary updated