Flexible, Corpus-Based Modelling of Human Plausibility Judgements

Sebastian Padó, Ulrike Padó, Katrin ErkEMNLP 2007

TL;DR

This paper introduces a flexible, corpus-driven model for predicting human judgments of verb-argument plausibility. By leveraging vector space representations of words and their syntactic roles, the model achieves high coverage and competitive accuracy, even outperforming deeper, resource-intensive models in some scenarios. The work highlights the effectiveness of combining shallow, corpus-based methods with deeper linguistic resources for improved natural language understanding.

What Problem Does This Paper Address?

Understanding how humans judge the plausibility of sentences is a fundamental question in psycholinguistics and a crucial capability for computational linguistics. For instance, humans readily understand that a "hunter" is a plausible agent of "shoot," but an implausible patient (i.e., being shot). This phenomenon is explained by "selectional preferences" – verbs tend to prefer certain types of arguments.

The challenge lies in computationally modeling these preferences. Previous approaches often relied on hand-crafted resources like WordNet or specialized semantic role annotations, which can limit their coverage (i.e., the range of words and structures they can handle) and pre-determine the kinds of generalizations they can make. This paper aims to develop a model that can predict human plausibility judgments using only a large, unannotated corpus, offering greater flexibility and broader applicability.

Methods

The core of the proposed model builds upon Erk's (2007) work, combining two key ideas:

  1. Selectional Preference Induction: The intuition is that if a new argument word is similar to words previously seen as plausible arguments for a given verb-relation pair, then the new argument is also likely plausible.
  2. Vector Space Models: Words are represented as vectors in a high-dimensional space, where similarity in meaning is captured by proximity in this space.

The novelty lies in how these are integrated:

  • Corpus-Driven Approach: Instead of relying on external resources, the model learns from a large corpus (the British National Corpus - BNC).
  • Syntactic Role Integration: The model uses dependency parsing to identify syntactic roles (like subject and object) and combines these with words to form basis elements for the vector space. This is a crucial step, as it allows the model to distinguish between, for example, "hunter" as an agent versus "hunter" as a patient, which simple word-based vector spaces struggle with.
  • Two Model Instantiations:
    • Unparsed Model: Uses word order to approximate syntactic roles (words to the left of a verb are subjects/agents, words to the right are objects/patients).
  • Parsed Model: Uses a dependency parser to explicitly identify subject and object relations.

The plausibility of a (verb, relation, argument) triple is then calculated by averaging the similarity between the argument's vector and the vectors of previously seen arguments for that verb-relation pair.

Datasets

  • Training Data: The British National Corpus (BNC), a large collection of text (over six million sentences).
  • Evaluation Datasets:
    • McRae et al. (1998) dataset: 100 data points, designed to be highly bimodal (arguments are either very plausible or very implausible for a given relation).
  • Pado dataset: 414 data points, constructed from corpus co-occurrences, showing a more even distribution of ratings.

Key Insights and Ideas

  • Syntax Matters for Plausibility: Explicitly modeling syntactic roles (agent/subject, patient/object) is critical for accurately predicting plausibility, far more so than relying on simple word co-occurrence or word order alone. This suggests that grammatical structure directly encodes semantic preferences.
  • Vector Space Topology is Key: The success of the model hinges on the "topology" of the vector space. It's not just about general word similarity, but about similarity in the context of specific syntactic roles. Using word-relation pairs as basis elements creates a more informative space than using just words.
  • Corpus-Driven is Viable: A completely corpus-driven approach can be a powerful alternative to resource-based methods, offering better coverage and competitive performance.
  • Complementary Strengths: Deeper, resource-based models (like Padó's) excel when their resources cover the data, but struggle with unseen cases. Shallow, corpus-driven models (like the proposed one) have broader coverage. Combining them can yield the best of both worlds.

Key Results

  • Parsed Models Outperform Unparsed: The dependency-parsed models consistently achieved significantly higher correlation with human judgments than their unparsed counterparts.
  • Parsed Cosine is Strong: The "Parsed Cosine" model (using dependency parsing and the Cosine similarity measure) achieved a Spearman's ρ of 0.218 on the McRae data and 0.479 on the Pado data, showing significant correlation.
  • Competitive with Deep Models: On the Pado dataset, the Parsed Cosine model achieved a correlation of 0.479, which was statistically indistinguishable from the deeper, FrameNet-based Padó model (0.515), despite the latter having a smaller coverage (97% vs. 98%).
  • Model Combination Boosts Performance: Combining the Padó model (for "FN-Seen" data points) with the Parsed Cosine model (for "FN-Unseen" data points) resulted in a correlation of 0.552, outperforming either model individually.

Limitations and Future Work

  • Approximating Relations: While dependency relations (subject/object) are good proxies for agent/patient, the paper acknowledges that approximating finer-grained semantic roles (beyond agent/patient) using only grammatical realisations might introduce noise. This is an open research question.
  • Model Combination Strategies: The paper explored a simple "naive backoff" combination strategy. More sophisticated methods for combining predictions from different models are suggested as future work.
  • Generalizing to More Relations: The current model focuses on agent/patient. Extending it to handle a wider range of semantic relations and their nuances is a natural next step.
  • Vector Space Construction: Further research into optimal vector space construction, including different basis elements and weighting schemes, could yield further improvements.

Summary updated

BibTeX
@inproceedings{pado2007flexible,
  author = {Sebastian Padó and Ulrike Padó and Katrin Erk},
  title = {{Flexible, Corpus-Based Modelling of Human Plausibility Judgements}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure