Improving the Use of Pseudo-Words for Evaluating Selectional Preferences

Nathanael Chambers, Daniel JurafskyACL 2010 · OpenReview

Abstract

This paper improves the use of pseudo-words as an evaluation framework for selectional preferences. While pseudo-words originally evaluated word sense disambiguation, they are now commonly used to evaluate selectional preferences. A selectional preference model ranks a set of possible arguments for a verb by their semantic fit to the verb. Pseudo-words serve as a proxy evaluation for these decisions. The evaluation takes an argument of a verb like drive (e.g. car), pairs it with an alternative word (e.g. car/rock), and asks a model to identify the original. This paper studies two main aspects of pseudoword creation that affect performance results. (1) Pseudo-word evaluations often evaluate only a subset of the words. We show that selectional preferences should instead be evaluated on the data in its entirety. (2) Different approaches to selecting partner words can produce overly optimistic evaluations. We offer suggestions to address these factors and present a simple baseline that outperforms the state-of-the-art by 13% absolute on a newspaper domain.

Introduction

In Natural Language Processing (NLP), understanding selectional preferences—the semantic fit between a verb and its arguments (e.g., eat "lunch" vs. eat "a rock")—is crucial. Due to the high cost of human-labeled datasets, researchers frequently employ pseudo-words as a proxy for evaluation. In this framework, a model is presented with a real verb-argument pair from a corpus and a "confounder" (a distractor word) and must correctly identify the original.

However, current evaluation methodologies suffer from two major flaws: the "Unseen" bias and the use of overly simplistic distractors. This paper investigates these issues and proposes a more robust evaluation framework.

Methodology

The study compares various methods for confounder selection and model architectures across several large-scale corpora, including the New York Times (NYT), Associated Press (APW), and the British National Corpus (BNC).

Confounder Selection Methods

  1. Random: Selecting any noun from the corpus (the standard approach).
  2. Buckets: Selecting a noun with a similar overall frequency.
  3. Neighbor: Selecting the noun with the closest possible frequency, creating the most challenging task.

Models Evaluated

  • Baseline: A simple conditional probability model ($P(noun|verb)$) based on co-occurrence counts in training data.
  • Web Baseline: Co-occurrence estimates derived from Google n-gram counts.
  • Erk (2007) Smoothing Model: A state-of-the-art model utilizing noun-noun similarity to estimate preferences for rare or unseen words.

The Impact of Training Scale on Unseen Arguments

A common practice in selectional preference evaluation is to focus solely on "unseen" word pairs—those not encountered during training. However, this ignores the reality of data distribution in large corpora. As training data increases, the percentage of unseen arguments in a test set drops significantly.

Percentage of unseen arguments decreases as training data size increases, reaching approximately 12% for the NYT corpus at 1.2 billion tokens.

In the NYT domain, once the training set reaches a sufficient size, nearly 90% of arguments in new documents have been seen before. Evaluating only on the remaining 10% provides a skewed view of a model's real-world utility.

Evaluating Model Performance and Confounder Selection

The choice of confounder significantly impacts reported performance. Using random distractors makes the task artificially easy, as the distractor often has a vastly different frequency or context than the original word. When switched to "nearest-neighbor" confounders, performance across all models drops significantly.

Comparison of model performance across different confounder selection methods. Note the significant drop in accuracy when moving from 'Random' to 'Neighbor' distractors.

As shown in the results, the simple Baseline model drops from 91.5% to 79.5% when moving from Random to Neighbor distractors. This "Neighbor" method provides a more conservative and reproducible benchmark, preventing the overly optimistic results prevalent in previous literature.

Key Results and Discussion

The most striking finding is that a simple conditional probability baseline outperforms complex, state-of-the-art smoothing models by approximately 13% absolute when evaluated on the full dataset. While similarity-based models like the Erk model are designed to handle sparsity, they often introduce noise that degrades performance on the "seen" data that constitutes the majority of the task.

As training data scales from 110 million to 1.2 billion tokens, the simple baseline continues to improve, whereas more complex models tend to stagnate. This suggests that for many practical applications, data scale and simple counting are more effective than complex smoothing algorithms.

Conclusion

This research highlights the importance of evaluating selectional preference models on entire datasets rather than just unseen subsets and using more rigorous confounder selection methods like "nearest-neighbor." Future work should focus on "backoff" strategies that leverage simple counts for frequent pairs and reserve complex smoothing only for the small fraction of truly novel arguments.

Summary updated

Abstract

This paper improves the use of pseudo-words as an evaluation framework for selectional preferences. While pseudo-words originally evaluated word sense disambiguation, they are now commonly used to evaluate selectional preferences. A selectional preference model ranks a set of possible arguments for a verb by their semantic fit to the verb. Pseudo-words serve as a proxy evaluation for these decisions. The evaluation takes an argument of a verb like drive (e.g. car), pairs it with an alternative word (e.g. car/rock), and asks a model to identify the original. This paper studies two main aspects of pseudoword creation that affect performance results. (1) Pseudo-word evaluations often evaluate only a subset of the words. We show that selectional preferences should instead be evaluated on the data in its entirety. (2) Different approaches to selecting partner words can produce overly optimistic evaluations. We offer suggestions to address these factors and present a simple baseline that outperforms the state-of-the-art by 13% absolute on a newspaper domain.
BibTeX
@inproceedings{chambers2010improving,
  author = {Nathanael Chambers and Daniel Jurafsky},
  title = {{Improving the Use of Pseudo-Words for Evaluating Selectional Preferences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=BkWNcjgO-H},
}

Loading related content...

Figure