TL;DR
This paper introduces SKIP, a neurosymbolic model that injects event semantic knowledge from VerbNet into neural networks for procedural text understanding. By leveraging explicit event structures, SKIP achieves state-of-the-art results on entity state tracking, outperforming previous models that relied solely on entity-centric knowledge.
What Problem Does This Paper Address?
Understanding procedural texts, like recipes or assembly instructions, requires tracking how entities change state over time. This task, known as "entity state tracking," is challenging due to the scarcity of annotated data and the reliance on human commonsense knowledge. Existing NLP models often struggle with this implicit knowledge. While some models incorporate entity-centric knowledge (e.g., from ConceptNet), this paper argues that explicit knowledge about events and their semantic structures is crucial for a deeper understanding of how states change. The authors aim to bridge this gap by integrating event semantic knowledge into neural models.
Methods
The paper presents two main contributions:
- SKIP (Semantic Knowledge In Procedural text understanding): This is a neurosymbolic model that builds upon the KOALA framework. The key innovation is the incorporation of event semantic knowledge extracted from VerbNet, a comprehensive English verb lexicon that encodes event structures into sequences of subevents.
- Knowledge Extraction: VerbNet's semantic parser is used to extract logical representations of subevents. These are then translated into natural language for neural processing.
- Neurosymbolic Integration: A BERT encoder is fine-tuned on these translated event semantics to create a "knowledge encoder." This knowledge encoder is then infused into a neural architecture (similar to KOALA's) using a "knowledge injector" module. This allows the model to selectively attend to relevant event knowledge during state and location prediction.
- Training: SKIP employs a multi-stage training process, first fine-tuning a text encoder on procedural texts and then training the knowledge encoder and the final model.
- LEXIS: A purely symbolic model designed to simulate human cognitive construal of events. It also leverages VerbNet subevents and semantic features, along with PropBank SRL for gaps. LEXIS is domain-independent and aims for explainability.
Datasets
- ProPara: Used for training and evaluating SKIP and LEXIS. This dataset contains human-authored paragraphs describing scientific processes and is annotated for entity state changes (Move, Create, Destroy) and entity locations.
- Recipes: Used for evaluating LEXIS. This dataset consists of human-annotated recipes with a wider variety of state change types (composition, cookedness, temperature, etc.).
Key Insights and Ideas
- Event Semantics Matter: The paper strongly suggests that explicit knowledge of event structures (what happens in a process, the temporal and causal relationships between subevents) is more impactful for procedural text understanding than entity-centric knowledge alone.
- VerbNet as a Rich Source: VerbNet, with its detailed semantic predicates and event decomposition, provides a valuable source of structured knowledge that can significantly improve NLP models.
- Neurosymbolic Synergy: Combining symbolic knowledge (VerbNet) with neural networks (BERT) allows models to leverage both explicit linguistic structures and the pattern recognition capabilities of deep learning.
- Domain Independence: The proposed approach, particularly with LEXIS, demonstrates a path towards models that can adapt to different domains and NLP tasks by relying on general linguistic resources.
- Knowledge Source Impact: For SKIP, fine-tuning the knowledge encoder solely on VerbNet event semantics yielded better results than using both VerbNet and ConceptNet, suggesting that the specific type and structure of knowledge are critical.
Key Results
- SKIP: Achieved state-of-the-art results on the ProPara dataset, outperforming the re-implemented KOALA baseline.
- LEXIS: Achieved a new state-of-the-art on the Recipes dataset (70.1% F1, a 11.7% improvement) and performed well on ProPara (55.6% F1).
- Ablation Studies: Showed that including both VerbNet and ConceptNet knowledge in SKIP's LM fine-tuning stage actually lowered performance compared to using only VerbNet. This highlights the importance of carefully selecting and integrating knowledge sources.
Limitations and Future Work
- VerbNet Dependency: The methodology heavily relies on the VerbNet lexicon and parser. Limitations in VerbNet's coverage (missing verbs or senses) and its inherent generalizations can impact the model's performance.
- Data Sparsity: VerbNet's training data is relatively small, which can lead to misclassifications.
- Recipe Dataset Challenges: Some state change types in the Recipes dataset are not directly represented in VerbNet, requiring the use of verb-specific features.
- Future Work: The authors suggest expanding SKIP to incorporate other linguistic knowledge sources, exploring its application to tasks like causal inference and textual entailment, and further improving the VerbNet lexicon and its coverage. They also plan to evaluate SKIP on the Recipes dataset in the future.
Summary updated