Abstract
In this paper we describe the SemEval-2010 Cross-Lingual Lexical Substitution task, which is based on the English Lexical Substitution task run at SemEval-2007. In the English version of the task, annotators and systems had to find an alternative substitute word or phrase for a target word in context. In this paper we propose a task where the target word and contexts will be in English, but the substitutes will be in Spanish. In this paper we provide background and motivation for the task and describe how the dataset will differ from a machine translation task and previous word sense disambiguation tasks based on parallel data. We describe the annotation process and how we anticipate scoring the system output. We finish with some ideas for participating systems.
Introduction
Traditional Word Sense Disambiguation (WSD) has historically relied on fixed "sense inventories" such as WordNet. However, these rigid categories often fail to capture the nuance of real-world usage, and linguists frequently debate the validity of specific inventories. The Cross-Lingual Lexical Substitution (CLLS) task addresses this by defining the "sense" of a word through its appropriate translations in a target language. This approach is particularly relevant for Machine Translation (MT), where the correct sense is often defined by how it should be rendered in another language.
Motivation and Problem Definition
The core challenge in CLLS is that meaning is often "fuzzy" and overlapping. For example, the English word interest might translate to interés or rédito in a financial context, but interés or participación when referring to a stake in a company. As shown in Figure 2, these translations reflect subtle overlaps in meaning that discrete sense boxes cannot easily represent.
Methodology and Annotation
The proposed framework for the CLLS task involves providing an English context and a target word, for which systems must find valid Spanish substitutes. Unlike standard WSD, there is no fixed list of possible answers (an "open inventory").
To create the gold standard, four native Spanish speakers from Mexico provided translations. Annotators were encouraged to provide as many valid translations as fit the context. They also identified if a target word was part of a multiword expression (e.g., "part and parcel"), which served as a filter for the task. Figure 1 illustrates the annotation interface where human experts evaluated target words like "severely" within their specific English sentence contexts.
Dataset and Evaluation
The evaluation dataset was constructed using sentences from the English Internet Corpus. It includes 201 target words across various parts of speech (nouns, verbs, adjectives, and adverbs), with 10 sentences provided for each word. Notably, the task provides no training data, forcing participants to utilize external resources such as parallel corpora or bilingual dictionaries.
Systems are evaluated using two primary metrics:
- "Best": Rewards the system for selecting the most frequent translation chosen by human annotators.
- "Out-of-Ten": Allows the system to provide up to 10 guesses. If any match the human gold standard, the system receives credit, rewarding the discovery of valid translations that might have been overlooked by some annotators.
Conclusion
By shifting the focus from rigid sense inventories to cross-lingual substitution, SemEval-2010 Task 2 provides a more flexible and application-oriented framework for evaluating lexical semantics. This task sets the stage for future systems to better handle the complexities of language translation and contextual nuance.
Summary updated