Abstract
Distributional data tells us that a man can swallow candy, but not that a man can swallow a paintball, since this is never attested. However both are physically plausible events. This paper introduces the task of semantic plausibility: recognizing plausible but possibly novel events. We present a new crowdsourced dataset of semantic plausibility judgments of single events such as "man swallow paintball". Simple models based on distributional representations perform poorly on this task, despite doing well on selection preference, but injecting manually elicited knowledge about entity properties provides a substantial performance boost. Our error analysis shows that our new dataset is a great testbed for semantic plausibility models: more sophisticated knowledge representation and propagation could address many of the remaining errors.
Introduction
Most Natural Language Processing (NLP) models rely on distributional data, learning word meanings based on co-occurrence. While effective for many tasks, this approach suffers from reporting bias: humans rarely document the obvious (e.g., "the sun is hot") or the bizarre-but-possible (e.g., "he swallowed a marble"). Consequently, a model might see "man swallows candy" frequently but never encounter "man swallows paintball," leading it to incorrectly conclude the latter is impossible.
This paper addresses the challenge of semantic plausibility—distinguishing between what is typical in text (selectional preference) and what is physically possible in the real world. While swallowing a paintball is atypical, it is plausible due to the object's physical properties (small size, low weight). In contrast, "man swallows desk" is physically impossible.
Methods
The researchers developed a hybrid neural architecture that combines standard distributional representations with structured world knowledge.
Model Architecture
The model consists of an ensemble neural network with two primary branches. One branch processes standard GloVe embeddings to capture distributional typicality (NN). The second branch processes physical features—such as size, weight, and rigidity—to capture world knowledge (WK).
World Knowledge Acquisition
To collect physical property data efficiently, the authors used a "Binning-by-Landmark" method. Annotators categorized objects into bins defined by familiar landmarks (e.g., for SIZE: watch, book, cat, person, jeep, stadium). This captures scale-dependent interactions better than simple relative comparisons; for instance, while a man is bigger than both a cat and an ant, he can only plausibly "hug" the cat. Six key dimensions were tracked: Sentience, Mass-Count, Phase, Size, Weight, and Rigidity.
Feature Propagation
Recognizing that manual annotation is expensive, the authors tested feature propagation using Ordinal Logistic Regression. This allows the model to estimate physical properties for unknown words based on their distributional context, treating properties like "size" as ordered scales rather than categorical labels.
Key Results
The inclusion of world knowledge significantly improved model performance on the semantic plausibility task.
A baseline neural network using only word embeddings achieved 68% accuracy. Injecting "Gold" (human-annotated) world knowledge (NN + WK-GOLD) boosted accuracy to 76%. Furthermore, the model remained robust even when using propagated features (NN + WK-PROP); with only 20% of the data annotated, it still achieved up to 74% accuracy. Error analysis revealed that 60% of the baseline's mistakes were directly attributable to missing information about size and weight.
Limitations and Future Work
Despite the improvements, several "long-tailed" physical attributes remain challenging. Current models lack features for:
- Edibility: Distinguishing between objects of similar size where only one is food (e.g., egg vs. cup).
- Hollowness: Understanding that a "purse" can contain "scissors" while a "rock" cannot.
- Agent Dexterity: Recognizing that a "girl" can "snatch a watch" but a "pig" cannot due to physical limitations.
Future work will focus on more sophisticated representations of object affordances and agent capabilities to bridge the remaining gap in machine common sense.
Summary updated