Cross-lingual Induction of Selectional Preferences with Bilingual Vector Spaces

Yves Peirsman, Sebastian PadóNAACL 2010

TL;DR

This paper introduces a novel cross-lingual method to learn selectional preferences for languages with limited linguistic resources. By leveraging bilingual vector spaces, they "translate" predicate-argument structures from a low-resource language into English, where rich parsed corpora are available, achieving performance that surpasses monolingual baselines.

What Problem Does This Paper Address?

Selectional preferences, which capture how well words fit into specific argument roles of a verb (e.g., "people" are good subjects for "shoot," while "animals" are good objects), are crucial for natural language understanding tasks. Building accurate models for these preferences typically requires large, syntactically parsed corpora. Unfortunately, such high-quality resources are only available for a handful of languages, leaving many languages without reliable selectional preference models. This paper aims to bridge this gap by enabling the induction of these preferences for resource-poor languages.

Methods

The core of the proposed method is a cross-lingual knowledge transfer approach using bilingual vector spaces.

  1. Bilingual Vector Space Construction: Instead of relying on parallel corpora or bilingual lexicons (which are often scarce or ambiguous for resource-poor languages), they construct a bilingual vector space from large, unparsed corpora in the source and target languages. This space represents words from both languages. Dimensions are formed by bilingual context word pairs (e.g., "secretly/heimlich"). The key idea is that if the context words surrounding a source word are translations of the context words surrounding a target word, their vector representations in this space will be similar. They use a bootstrapping process: starting with a small set of cognates (words that are similar in spelling and meaning across languages), they iteratively identify nearest neighbors in the vector space and add them as new dimensions, refining the bilingual mapping.
  2. Cross-lingual Translation of Triples: Once the bilingual vector space is built, they can "translate" source language words (predicates and head words) into their closest target language (English) equivalents within this space. This translation is not necessarily a literal one; it identifies semantically similar words.
  3. Selectional Preference Induction: The translated triples (predicate, argument position, head word) are then fed into a well-established English selectional preference model (based on distributional similarity). This allows them to predict plausibility scores for the source language.

They explore two experimental setups:

  • Experiment 1 (Syntax-less): Assumes argument positions (subject, object) are directly transferable between languages.
  • Experiment 2 (Syntax-aware): Attempts to learn a mapping between source and target argument positions, acknowledging that syntactic realization can differ (e.g., a German prepositional object might correspond to an English direct object). This requires some syntactic information from the source language.

Datasets

  • Target Language (English): The British National Corpus (BNC) was used for training the English selectional preference model.
  • Source Languages:
    • German: The Huge German Corpus (HGC) was used for building the bilingual vector space. Plausibility judgments for evaluation were taken from Brockmann (2002), based on the TiGer corpus.
  • Spanish: A corpus of approximately 100 million words from Spanish newspapers was used for the bilingual vector space. Plausibility judgments for evaluation were collected via Amazon Mechanical Turk (AMT) from native Spanish speakers. Two syntactically analyzed Spanish corpora, AnCora and Encarta, were used for training monolingual models and in the syntax-aware experiment.

Key Insights and Ideas

  • Bilingual Vector Spaces as a Bridge: The paper demonstrates that bilingual vector spaces, constructed without parallel data, are a powerful mechanism for transferring linguistic knowledge across languages, even for complex tasks like selectional preference induction.
  • Semantic Similarity Over Literal Translation: The success of the method hinges on identifying semantically related words, not just literal translations. This is crucial because words with similar semantic roles might be expressed differently syntactically.
  • Argument Position Mapping is Key: While a simple transfer of argument positions (Experiment 1) yields good results, explicitly learning mappings between source and target argument positions (Experiment 2) significantly improves performance, especially when syntactic structures differ.
  • Noisy Data Can Be Useful: Even "noisy" syntactic data from the source language can be beneficial, suggesting that resource-poor languages might still contribute valuable information if their data can be integrated effectively.
  • Cross-lingual Transfer Outperforms Monolingual Baselines: The cross-lingual approach, even with minimal source language syntactic information, achieves competitive or superior results compared to monolingual models trained on limited or noisy data.

Key Results

  • German: The cross-lingual model (Experiment 1) achieved significant correlations with human judgments, outperforming both ontological and distributional monolingual baselines. Experiment 2 further improved these results, with correlations reaching up to .48 for the complete dataset.
  • Spanish: The Spanish results were more varied. Experiment 1 showed significant correlations for subjects but not for objects. Experiment 2 showed improvement, particularly for objects, with the complete dataset correlation increasing from .36 to .47. The performance on the Encarta corpus remained disappointing, highlighting the impact of data quality.
  • Argument Position Transfer: The syntax-aware model (Experiment 2) demonstrated improvements by learning mappings like German "stagnieren" (stagnate) SUBJ -> English "boost" DOBJ, showing that the model can recover non-identity mappings.
  • Limited Syntactic Data Suffices: Figure 3 shows that most of the benefit from syntactic information is gained from the first few occurrences of predicate-argument pairs, suggesting that even a small amount of targeted syntactic annotation can substantially improve the cross-lingual model.

Limitations and Future Work

  • Limitations:
    • The method is sensitive to the quality of the bilingual vector space and the target language model.
  • "False friends" and differences in register or domain between corpora can lead to incorrect translations and affect plausibility estimates.
  • The Spanish results highlight challenges with significant syntactic differences between languages (e.g., animate direct objects marked by prepositions in Spanish).
  • The model's performance can be affected by data sparsity for infrequent words.
  • Future Work:
    • Developing truly bilingual models where source language information actively improves the target language model.
  • Extending the cross-lingual mapping for argument positions to handle more complex, predicate-dependent mappings (e.g., based on animacy).
  • Investigating how to better handle polysemy and idiomatic expressions.
  • Exploring the transferability to languages with greater linguistic distance from English.

Summary updated

BibTeX
@inproceedings{peirsman2010cross,
  author = {Yves Peirsman and Sebastian Padó},
  title = {{Cross-lingual Induction of Selectional Preferences with Bilingual Vector Spaces}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure