Abstract
Computational semantic models rely heavily on human-annotated gold standards for evaluation. However, such resources have historically been concentrated on English and a few other high-resource languages. This paper presents ViCon and ViSim-400, the first comprehensive datasets designed to evaluate semantic similarity and relatedness in Vietnamese. ViCon targets lexical contrast across 1,400 word pairs, while ViSim-400 provides graded similarity scores for 400 pairs across various semantic relations. Our analysis shows high inter-annotator agreement and demonstrates that these datasets effectively benchmark the performance of diverse word embedding models in a linguistically unique, low-resource context.
Introduction
In Natural Language Processing (NLP), word embeddings are fundamental for representing lexical meaning. To validate these representations, researchers compare model outputs against human judgments of semantic similarity (e.g., cup vs. mug) and relatedness (e.g., cup vs. coffee). While datasets like SimLex-999 have become standard for English, Vietnamese—an isolating language with no morphological marking for case, gender, or tense—has lacked equivalent benchmarks. This gap hinders the development and evaluation of NLP tools for the millions of Vietnamese speakers worldwide.
Dataset Construction: ViCon and ViSim-400
We introduce two distinct benchmarks to address different aspects of semantic understanding:
- ViCon (Lexical Contrast): This dataset contains 1,400 word pairs (600 adjectives, 400 nouns, and 400 verbs) categorized as either synonyms or antonyms. Pairs were sourced from the Vietnamese Computational Lexicon (VCL) and Vietnamese WordNet (VWN).
- ViSim-400 (Semantic Relatedness): This dataset captures the graded nature of semantic relations. It consists of 400 word pairs covering synonymy, antonymy, hypernymy, co-hyponymy, and meronymy, alongside unrelated pairs.
For ViSim-400, 200 native speakers rated pairs on a scale of 0 (unrelated) to 6 (highly similar). To ensure data quality, "checkpoint" questions were used to filter out inattentive participants. The resulting scores show that synonyms are consistently rated as the most similar, while antonyms and unrelated pairs are correctly identified as the most dissimilar across all word classes.
Human Agreement and Data Reliability
A critical finding of this work is the high reliability of the Vietnamese annotations. The inter-annotator agreement (IAA) for ViSim-400 reached a Spearman correlation of $\rho = 0.79$. Notably, this is higher than the agreement reported for the English SimLex-999 ($\rho = 0.67$), suggesting that the Vietnamese dataset provides an exceptionally stable gold standard. Consistent with linguistic theory, agreement was slightly higher for nouns than for more abstract categories like verbs and adjectives.
Experimental Evaluation
To validate the datasets, we tested several semantic models, including traditional co-occurrence models (PPMI) and neural architectures like Skip-gram (SGNS) and dLCE (which incorporates lexical contrast).
The results indicate that Vietnamese models follow performance patterns similar to those observed in English. The dLCE model significantly outperformed standard Skip-gram models, achieving a correlation of 0.62 on ViSim-400 compared to 0.37 for SGNS. Furthermore, on the ViCon dataset, dLCE demonstrated a high capacity for distinguishing synonyms from antonyms, reaching an Area Under Curve (AUC) of up to 0.98 for adjectives.
Conclusion
ViCon and ViSim-400 provide the first robust benchmarks for Vietnamese semantic evaluation. By highlighting the distinction between similarity and relatedness and providing high-quality human judgments, these datasets enable more accurate assessment of Vietnamese language models. Future work will involve expanding these resources to include multi-word expressions and leveraging them to fine-tune large-scale transformer models specifically for the Vietnamese context.
Summary updated