BCWS: Bilingual Contextual Word Similarity

Ta-Chung Chi, Ching-Yen Shih, Yun-Nung ChenarXiv 2018 · OpenReview

TL;DR

This paper introduces BCWS, the first dataset for evaluating bilingual contextual word similarity between English and Chinese. By providing word pairs within their sentence contexts and human-annotated similarity scores, BCWS aims to advance cross-lingual understanding beyond simple word-level similarity, tackling the challenge of polysemy across languages.

What Problem Does This Paper Address?

Current natural language processing (NLP) models excel at capturing word meanings through embeddings, but often struggle with polysemy – words having multiple meanings. While datasets like SCWS and WiC have been developed to address this in English, there's a significant gap in evaluating how well models handle word senses in a bilingual context, especially for languages as different as English and Chinese. Existing cross-lingual similarity datasets lack sentence contexts, making it difficult to assess sense-specific similarity. This paper tackles this by creating a dataset that bridges this gap, enabling the development of more nuanced multilingual AI.

Methods

The authors construct the BCWS dataset through a multi-step process:

  1. Chinese Multi-Sense Word Extraction: They start by identifying frequent, polysemous Chinese words (nouns, adjectives, verbs) from Wikipedia and Chinese WordNet, filtering out words with too few or too many senses to ensure meaningful evaluation.
  2. English Candidate Word Extraction: Using BabelNet, a multilingual lexical database, they find English counterparts for the selected Chinese words.
  3. Enriching Semantic Relationship: To capture richer semantic connections, they augment the English candidates with hyponyms, hypernyms, and holonyms. They then sample a specific number of English words to form bilingual word pairs.
  4. Adding Contextual Information: For each bilingual word pair, they extract relevant sentences from Chinese WordNet and English Wikipedia, ensuring the target words are present and tagged with their part-of-speech.
  5. Human Labeling: Crucially, they recruit 11 native Chinese speakers with strong English proficiency to annotate the semantic similarity of these word pairs within their contexts, using a 0-10 scale.

Datasets

  • BCWS (Bilingual Contextual Word Similarity): This is the primary contribution of the paper. It comprises 2,091 English-Chinese word pairs, each with corresponding sentence contexts and human-annotated similarity scores. It's designed to evaluate cross-lingual sense embeddings.
  • Chinese Wikipedia dump: Used for extracting frequent Chinese words.
  • Chinese WordNet: Used for identifying polysemous Chinese words and extracting Chinese sentences.
  • BabelNet: Utilized as a bilingual dictionary to find English counterparts for Chinese words.
  • English Wikipedia dump: Used for extracting English sentences containing target words.
  • UM-corpus: A large parallel English-Chinese corpus used for training baseline models.

Key Insights and Ideas

  • The Importance of Context for Bilingual Similarity: The paper strongly emphasizes that simply comparing word embeddings isn't enough for bilingual similarity. Sentence context is vital for disambiguating word senses, especially across languages.
  • High Inter-Annotator Consistency is Achievable: The BCWS dataset boasts a significantly higher inter-annotator agreement (0.83) compared to previous English-only contextual word similarity datasets (e.g., SCWS at 0.52). This suggests that with careful annotation guidelines and qualified annotators, reliable cross-lingual sense similarity judgments can be obtained.
  • Bridging Monolingual and Multilingual Understanding: The creation of BCWS is framed as a crucial step towards moving AI from understanding a single language to understanding multiple languages in a more nuanced, sense-aware manner.

Key Results

  • The BCWS dataset contains 2,091 English-Chinese word pairs with contextual information and human annotations.
  • The inter-annotator agreement for BCWS is 0.83, which is substantially higher than the 0.52 reported for the SCWS dataset.
  • Baseline experiments show that existing word-level and bilingual embedding models perform relatively poorly on BCWS (e.g., a word-level skip-gram model achieved 0.49 Spearman's rank correlation), indicating that BCWS presents a challenging benchmark and offers significant room for improvement.

Limitations and Future Work

  • Limited Scope of Baseline Models: The paper acknowledges that the baseline experiments are limited and that more advanced cross-lingual sense embedding models could be explored.
  • Potential for Further Improvement: While BCWS has high consistency, the authors suggest it provides a "large room for improvement" for future research, implying that even higher human performance might be achievable with more refined annotation or larger datasets.
  • Future Direction: The authors envision BCWS as a catalyst for exploring new directions in cross-lingual word and sense embeddings, ultimately contributing to more robust multilingual AI systems.

Summary updated

Abstract

This paper introduces the first dataset for evaluating English-Chinese Bilingual Contextual Word Similarity, namely BCWS (https://github.com/MiuLab/BCWS). The dataset consists of 2,091 English-Chinese word pairs with the corresponding sentential contexts and their similarity scores annotated by the human. Our annotated dataset has higher consistency compared to other similar datasets. We establish several baselines for the bilingual embedding task to benchmark the experiments. Modeling cross-lingual sense representations as provided in this dataset has the potential of moving artificial intelligence from monolingual understanding towards multilingual understanding.
BibTeX
@misc{chi2018bcws,
  author = {Ta-Chung Chi and Ching-Yen Shih and Yun-Nung Chen},
  title = {{BCWS: Bilingual Contextual Word Similarity}},
  year = {2018},
  eprint = {1810.08951},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.08951},
}
Figure