Abstract
Distributional data tells us that a man can swallow candy, but not that a man can swallow a paintball, since this is never attested. However both are physically plausible events. This paper introduces the task of semantic plausibility: recognizing plausible but possibly novel events. We present a new crowdsourced dataset of semantic plausibility judgments of single events such as "man swallow paintball". Simple models based on distributional representations perform poorly on this task, despite doing well on selection preference, but injecting manually elicited knowledge about entity properties provides a substantial performance boost. Our error analysis shows that our new dataset is a great testbed for semantic plausibility models: more sophisticated knowledge representation and propagation could address many of the remaining errors.
Introduction
In Natural Language Processing (NLP), models are traditionally trained on massive text corpora to learn selectional preference—the likelihood of word combinations based on frequency. For instance, "man eats cake" is common, while "man eats paintball" is rare. However, these models often fail to distinguish between what is merely rare and what is physically impossible. While a man rarely swallows a paintball, it is physically plausible. In contrast, a man swallowing a desk is physically implausible. Standard distributional models treat both as equally unlikely because neither appears frequently in text. This paper addresses this gap by introducing the task of semantic plausibility: recognizing if an event is physically possible, regardless of its textual frequency.
Methods
The authors propose a hybrid approach that ensembles distributional signals with structured physical world knowledge.
World Knowledge Elicitation
To scale the collection of physical properties, the researchers used a Binning-by-Landmark approach. Instead of expensive pairwise comparisons, annotators placed 450 common nouns into bins defined by landmarks (e.g., for SIZE: watch < book < cat < person < jeep < stadium). Six dimensions were captured: Sentience, Mass-Count, Phase, Size, Weight, and Rigidity.
Model Architecture (NN + WK)
The architecture consists of two main components:
- Neural Network (NN): A standard network using 300D GloVe embeddings to capture distributional data.
- World Knowledge (WK) Network: A dedicated branch that processes the physical attributes of the subject and object.
To handle words outside the initial 450-noun set, the authors used Ordinal Logistic Regression to propagate features to unknown words based on a small seed set (5-20% of the data).
Results
The researchers evaluated their approach on a new crowdsourced dataset of 3,062 Subject-Verb-Object (SVO) triples. The results demonstrate that injecting world knowledge significantly improves the model's ability to judge plausibility.
As shown in the table above, the distributional baseline (NN) achieved 68% accuracy. Adding gold-standard world knowledge (NN + WK-GOLD) increased this to 76%. Even when using propagated features (NN + WK-PROP), the model maintained a high accuracy of 74%, proving that physical properties can be effectively generalized to unseen vocabulary. Error analysis showed that 60% of the errors fixed by the world-knowledge model were related to Size and Weight.
Discussion and Conclusion
The primary challenge identified is the "Obviousness Gap": humans rarely document basic physical facts (e.g., "desks are larger than people"), leaving distributional models "blind" to physical constraints. While the current model successfully uses six core dimensions, error analysis reveals a "long tail" of more nuanced attributes like edibility, containment (hollow vs. solid), and anatomical dexterity. Future work will explore using computer vision and more advanced knowledge propagation to capture these complex physical dimensions, moving closer to true semantic understanding.
Summary updated