Abstract
Using low dimensional vector space to represent words has been very effective in many NLP tasks. However, it doesn't work well when faced with the problem of rare and unseen words. In this paper, we propose to leverage the knowledge in semantic dictionary in combination with some morphological information to build an enhanced vector space. We get an improvement of 2.3% over the state-of-the-art Heidel Time system in temporal expression recognition, and obtain a large gain in other name entity recognition (NER) tasks. The semantic dictionary Hownet alone also shows promising results in computing lexical similarity.
Introduction
Distributed word embeddings, such as Word2Vec, have revolutionized NLP by representing words as dense, continuous vectors. These models typically rely on co-occurrence statistics derived from massive corpora. However, they suffer from a significant "sparsity problem": rare words with few occurrences or unseen words (out-of-vocabulary) often receive poor-quality or non-existent representations. This issue is particularly acute in Chinese, where new words are frequently synthesized from a stable set of characters and underlying semantic units called "sememes."
Methodology
To address the sparsity problem, we propose a multi-component architecture that integrates structured prior knowledge from the HowNet semantic dictionary with morphological features. The final "complete vector space" is formed by integrating four distinct types of information:
- HowNet Semantic Knowledge: HowNet defines words through a finite set of approximately 1,500 sememes. We train embeddings for these sememes by substituting words in a corpus with their corresponding sememe definitions. A word's vector is then derived as the algebraic sum of its constituent sememes' vectors.
- Morphological Similarity: For rare words, we identify "similar words" in the known vocabulary using Longest Common Substring (LCS) and edit distance. A "Similar word vector" is generated via a weighted average of the top-5 candidates.
- Character-Level Embedding: In Chinese, the final character of a noun often serves as the semantic "head." We incorporate character-level embeddings of the word's final character to capture this structural information.
- Standard Word2Vec: We retain standard context-based embeddings as a baseline component.
As illustrated in the architecture, the standard and similar-word vectors are first combined before being concatenated with HowNet and character-level embeddings to form the final representation used for downstream classification.
Experimental Setup
The proposed model was evaluated across three distinct tasks:
- Lexical Similarity: A human-judged dataset of 40 word pairs was used to correlate model outputs with human intuition.
- Temporal Expression Recognition: Evaluated on the Tempeval-2 (Chinese) dataset, which includes 931 training and 345 test sentences focused on identifying dates, durations, and times.
- Named Entity Recognition (NER): Tested on a corpus from the People’s Daily (1998) for identifying person names (NR) and location names (NS).
Key Results
The integration of semantic and morphological knowledge yielded significant performance gains over baseline models and established rule-based systems.
Temporal Tagging
The "Final Model" achieved an F-score of 89.6%, representing a 2.3% improvement over the previous state-of-the-art rule-based system, Heidel Time (87.3%).
Named Entity Recognition
On the People’s Daily NER task, the enhanced embeddings provided a substantial boost. For location names (NS), the F-score increased significantly from 72.9 (standard Word2Vec) to 82.2 (Final Model).
Human Correlation
The HowNet-based embeddings achieved a Spearman correlation of 56.6 with human similarity rankings, outperforming previous computational methods based on the same dictionary (which scored 38.0).
Discussion and Conclusion
Our findings suggest that sememes can act as a "universal language" for word representation. Even for rare words, the underlying sememes are often common, allowing for the construction of robust proxy vectors. Furthermore, the simple additive property of sememe vectors proved surprisingly effective at capturing core semantic meaning.
While powerful, the approach has limitations. Summing sememes can lead to a loss of nuance or increased ambiguity. Additionally, data skew in specific tasks, such as temporal tagging, remains a challenge for machine learning classifiers. Future work will explore automated sememe embedding construction and integration with more advanced neural architectures.
Summary updated