TL;DR
This paper introduces a new NLP task: learning the "prototypical function" of human-made physical objects. They propose a transformer-based model that leverages artifact and frame definitions to achieve state-of-the-art performance, outperforming baselines that rely on knowledge bases or simple co-occurrence.
What Problem Does This Paper Address?
When we encounter physical objects in language, we often infer their intended use without explicit mention. For example, hearing "She borrowed the book" implies she'll read it, and "He killed the mayor with a gun" implies a shooting. This "commonsense knowledge" about an object's primary function is crucial for natural language understanding. However, current NLP systems struggle to capture this nuanced understanding. This paper aims to bridge this gap by developing methods to learn and represent these prototypical functions for human-made artifacts.
Methods
The authors tackle this problem by framing it as a multiclass classification task, where the goal is to assign a single prototypical function (represented by a FrameNet frame) to each physical artifact. They explore several approaches:
- Baselines: They first evaluate existing resources like ConceptNet and COMET, which contain "UsedFor" relations, and a co-occurrence baseline that identifies verbs frequently appearing with artifacts.
- Masked Language Model (MLM) Baseline: They leverage BERT's masked language modeling capabilities by using sentence templates (e.g., "A [artifact] can be used to [MASK]") to predict likely actions associated with an artifact.
- Learning from Definitions (PFdef): This approach fine-tunes BERT using both the dictionary definitions of artifacts and the definitions of FrameNet frames. The hypothesis is that semantic similarities between these definitions can reveal prototypical functions.
- Learning from Definitions plus Masked Patterns (PFdef+mask): This is their proposed state-of-the-art model, which combines the strengths of the PFdef model and the MLM approach. It concatenates representations from both definition-based and masked-pattern-based processing to jointly learn the optimal function prediction.
Datasets
- Gold Standard Dataset: The authors created a new dataset consisting of 938 physical artifacts. Each artifact is manually labeled with one of 42 selected FrameNet frames representing its most prototypical function, or labeled as "None" if no appropriate frame was found. This dataset was created by annotators using WordNet definitions and detailed guidelines, achieving a good inter-annotator agreement (average Cohen's kappa of 0.72).
- WordNet: Used to extract initial lists of potential artifacts.
- FrameNet: Used to represent prototypical functions via its semantic frames and lexical units.
- English Wikipedia Corpus: Used for the co-occurrence baseline.
Key Insights and Ideas
- Prototypicality is Key: The paper strongly emphasizes that while objects can have multiple uses, identifying the prototypical use is crucial for commonsense understanding.
- Definitions Hold Power: The success of the PFdef model highlights that rich information about an artifact's function is often embedded within its dictionary definition and the definitions of related frames.
- Masked Patterns Capture Nuance: The MLM approach, while not as strong as definition-based methods alone, proves valuable in capturing certain functional relationships that definitions might miss.
- Combining Approaches is Optimal: The PFdef+mask model demonstrates that integrating information from both definitions and masked language modeling leads to the best performance, suggesting a synergistic relationship between these methods.
- FrameNet is a Suitable Representation: The choice of FrameNet frames as a canonical representation for functions is effective, allowing for generalization across similar uses and avoiding issues with varied phrasing.
Key Results
The proposed PFdef+mask model achieved the highest performance, with an accuracy of 76.8% and an F1 score of 62.4%. This significantly outperforms all baseline models:
- Baselines: ConceptNet (F1: 16.4%), Co-occurrence (F1: 19.9%), COMET (F1: 28.2%).
- MLM Baseline: F1: 35.4%.
- PFdef Model: F1: 59.3%.
The PFdef+mask model showed improvements in both precision and recall compared to the individual PFdef model. The study also found that frames with more training instances (high-frequency frames) generally led to better performance.
Limitations and Future Work
- Limitations:
- The dataset excluded food items, as many are naturally occurring.
- The "None" category captured artifacts that are materials (e.g., fabric) or used for general purposes (e.g., computer, laptop), suggesting limitations in the current frame set.
- The MLM baseline sometimes produced tangential or random answers.
- Future Work:
- The authors suggest that this learned functional knowledge could be integrated into semantic parsing to improve sentence-level understanding.
- They also envision using this knowledge for situation recognition in images.
- Further work could explore expanding the set of frames and artifacts, particularly for categories like materials and general-purpose objects.
Summary updated