TL;DR
This paper explores using translations as a proxy for word meaning to determine if a word has the same sense in two different sentences. The authors propose methods that leverage multilingual translations and contextual embeddings, showing that incorporating more languages improves accuracy on the Word-in-Context (WiC) task.
What Problem Does This Paper Address?
The core challenge is Word-in-Context (WiC) disambiguation: determining if a word in two different sentences carries the same meaning. This is crucial for many Natural Language Processing (NLP) tasks like machine translation, information retrieval, and question answering. While previous work has focused on contextual embeddings or traditional word sense disambiguation (WSD) systems, this paper investigates a novel approach: using translations as a signal for semantic similarity. The intuition is that if two senses of a word have different meanings, they are likely to translate differently into other languages. This approach aims to be more explainable by grounding decisions in linguistic phenomena.
Methods
The authors propose four methods, all relying on identifying lexical translations of the focus word in translated sentences.
- IDENT: A baseline that simply checks if the translations of the focus word are identical in terms of lemma and part-of-speech across all target languages.
- CVAL: Based on the theoretical concept of "multi-synset intersection." It checks if there's at least one common semantic concept (represented by a BabelNet multi-synset) that encompasses the focus word and its translations.
- SUB: This method uses a "synonymy check" implemented with dense contextualized embeddings (specifically, BERT). It applies this check to pairs of translated sentences, creating new sentences where one translation is substituted for another. If the synonymy check passes for all target languages where translations are available, the method returns TRUE.
- CSUB: Combines CVAL and SUB. It uses the multi-synset intersection idea but defers to the synonymy check (SUB) unless the intersection is empty (|C|=0), in which case it immediately returns FALSE.
The core of the "synonymy check" involves generating contextualized embeddings for word tokens and comparing their cosine similarity. If the similarity exceeds a threshold, it's considered evidence that substituting one word for another doesn't change the sentence's meaning.
Datasets
The paper uses the SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context (MCL-WiC) dataset for evaluation. Specifically, they focus on the monolingual English variant of the task. They do not use training data as their methods are unsupervised.
Key Insights and Ideas
- Translations as a Semantic Signal: The fundamental idea that different word senses often have different translations is empirically supported. This provides a linguistically grounded way to approach WiC.
- Multilingualism Boosts Accuracy: Incorporating translations from multiple languages consistently improves performance across most proposed methods, suggesting that a richer set of translations provides a more robust signal of semantic equivalence.
- Contextual Embeddings for Synonymy: Using BERT embeddings to implement a substitution test for synonymy is an effective way to capture subtle semantic similarities between words in context.
- Simplicity Can Be Effective: The IDENT baseline, despite its simplicity, performs surprisingly well, highlighting the power of direct translation comparison.
Key Results
On the English development set:
- The SUB method, which uses BERT-based synonymy checks across translated sentences, achieved the best performance, especially when using all three translation languages (French, Italian, Russian), reaching 60.8% accuracy.
- The IDENT baseline also performed strongly, achieving 59.7% accuracy with all languages.
- Methods relying heavily on multi-synset intersection (CVAL, CSUB) showed lower accuracy and tended to degrade as more languages were added.
On the English test set:
- The SUB method again led with 63.2% accuracy using all three languages.
- The IDENT baseline also performed well, reaching 61.1% accuracy.
- Notably, Russian translations consistently yielded better results than French or Italian for most methods.
- The best performing method (SUB) showed a substantial reduction in false positives when using multiple languages, at a potential slight increase in false negatives, but with an overall higher accuracy.
Limitations and Future Work
- Translation Quality: The methods are sensitive to the quality of machine translation. Errors or non-literal translations can lead to incorrect decisions (false negatives or positives).
- Ambiguous Translations: Sometimes, distinct senses of a word might translate to the same word in another language, leading to false positives. Conversely, distinct but synonymous translations can sometimes lead to false positives if they pass the synonymy check.
- Limited Language Coverage: While multilingualism is beneficial, the choice of languages (French, Italian, Russian) was constrained by the shared task and BabelNet coverage. Future work could explore a wider range of languages.
- Supervised Competition: The authors acknowledge their methods are not competitive with state-of-the-art supervised systems but see their work as a strong foundation for future research in contextual semantic analysis.
- Explainability: While aiming for explainability, the reliance on BERT embeddings in SUB and CSUB might reduce direct interpretability compared to IDENT.
Future work could involve exploring more sophisticated alignment techniques, integrating richer semantic resources, and investigating the impact of even more languages.
Summary updated