SemEval-2010 Task 2: Cross-Lingual Lexical Substitution

Ravi Som Sinha, Diana McCarthy, Rada Mihalcea

Abstract

In this paper we describe the SemEval-2010 Cross-Lingual Lexical Substitution task, which is based on the English Lexical Substitution task run at SemEval-2007. In the English version of the task, annotators and systems had to find an alternative substitute word or phrase for a target word in context. In this paper we propose a task where the target word and contexts will be in English, but the substitutes will be in Spanish. In this paper we provide background and motivation for the task and describe how the dataset will differ from a machine translation task and previous word sense disambiguation tasks based on parallel data. We describe the annotation process and how we anticipate scoring the system output. We finish with some ideas for participating systems.

Introduction

Traditional Word Sense Disambiguation (WSD) has historically relied on fixed "sense inventories" such as WordNet. However, these rigid categories often fail to capture the nuance of real-world usage, and linguists frequently debate the validity of specific inventories. The Cross-Lingual Lexical Substitution (CLLS) task addresses this by defining the "sense" of a word through its appropriate translations in a target language. This approach is particularly relevant for Machine Translation (MT), where the correct sense is often defined by how it should be rendered in another language.

Motivation and Problem Definition

The core challenge in CLLS is that meaning is often "fuzzy" and overlapping. For example, the English word interest might translate to interés or rédito in a financial context, but interés or participación when referring to a stake in a company. As shown in Figure 2, these translations reflect subtle overlaps in meaning that discrete sense boxes cannot easily represent.

Comparison of WordNet senses and their corresponding Spanish translations for the word 'interest'.

Methodology and Annotation

The proposed framework for the CLLS task involves providing an English context and a target word, for which systems must find valid Spanish substitutes. Unlike standard WSD, there is no fixed list of possible answers (an "open inventory").

To create the gold standard, four native Spanish speakers from Mexico provided translations. Annotators were encouraged to provide as many valid translations as fit the context. They also identified if a target word was part of a multiword expression (e.g., "part and parcel"), which served as a filter for the task. Figure 1 illustrates the annotation interface where human experts evaluated target words like "severely" within their specific English sentence contexts.

The annotation interface used by native Spanish speakers to provide contextual translations for English target words.

Dataset and Evaluation

The evaluation dataset was constructed using sentences from the English Internet Corpus. It includes 201 target words across various parts of speech (nouns, verbs, adjectives, and adverbs), with 10 sentences provided for each word. Notably, the task provides no training data, forcing participants to utilize external resources such as parallel corpora or bilingual dictionaries.

Systems are evaluated using two primary metrics:

  • "Best": Rewards the system for selecting the most frequent translation chosen by human annotators.
  • "Out-of-Ten": Allows the system to provide up to 10 guesses. If any match the human gold standard, the system receives credit, rewarding the discovery of valid translations that might have been overlooked by some annotators.

Conclusion

By shifting the focus from rigid sense inventories to cross-lingual substitution, SemEval-2010 Task 2 provides a more flexible and application-oriented framework for evaluating lexical semantics. This task sets the stage for future systems to better handle the complexities of language translation and contextual nuance.

Summary updated

Abstract

In this paper we describe the SemEval-2010 Cross-Lingual Lexical Substitution task, which is based on the English Lexical Substitution task run at SemEval-2007. In the English version of the task, annotators and systems had to find an alternative substitute word or phrase for a target word in context. In this paper we propose a task where the target word and contexts will be in English, but the substitutes will be in Spanish. In this paper we provide background and motivation for the task and describe how the dataset will differ from a machine translation task and previous word sense disambiguation tasks based on parallel data. We describe the annotation process and how we anticipate scoring the system output. We finish with some ideas for participating systems.
BibTeX
@inproceedings{sinha2010semeval,
  author = {Ravi Som Sinha and Diana McCarthy and Rada Mihalcea},
  title = {{SemEval-2010 Task 2: Cross-Lingual Lexical Substitution}},
  booktitle = {SemEval @ ACL},
  year = {2010},
}
Figure