Semantically-informed Hierarchical Event Modeling

Shubhashis Roy Dipta, Mehdi Rezaee, Francis FerraroACL 2023

TL;DR

This paper introduces a novel hierarchical event modeling framework that leverages both structural and semantic relationships between events. By integrating structured ontological knowledge (like FrameNet) into a multi-layer neural network, the model achieves state-of-the-art performance in predicting future events, understanding event sequences, and representing event similarity, even with limited semantic information.

What Problem Does This Paper Address?

Understanding complex events often involves recognizing their hierarchical nature – how individual events contribute to larger goals or scenarios. However, this hierarchical structure is rarely explicitly labeled in text, making it challenging for models to learn. Existing approaches either focus on structural hierarchy (e.g., using latent variables) or semantic richness (e.g., using ontologies), but struggle to combine both effectively. Furthermore, real-world event data often has incomplete or noisy semantic annotations, posing a significant challenge for models that rely heavily on such information. This paper aims to bridge this gap by developing a model that can learn hierarchical event representations by jointly leveraging structural and semantic information, while being robust to missing or imperfect semantic data.

Methods

The core of the proposed method is a Semantically-informed Hierarchical Event Modeling (SHEM) framework, built upon a hierarchical encoder-decoder architecture. It consists of two main layers:

  • Base Layer: This layer takes a sequence of observed events (represented as tuples of predicate, subject, object, etc.) and uses a semi-supervised approach to infer a sequence of semantic frames. It's designed to handle partially observed frames, meaning some event frames might be masked during training to simulate real-world data scarcity.
  • Compression Layer: This layer takes the inferred semantic frames from the base layer and further compresses them into more abstract concepts. This compression is guided by structured ontological knowledge, specifically using FrameNet's frame relations (like inheritance, causation, etc.) to identify semantically similar frames and group them into higher-level abstract frames.

The model uses a Gumbel-Softmax distribution for sampling discrete latent variables (frames) within the neural network, allowing for differentiable learning. The training objective involves a combination of reconstruction loss (to regenerate the input event sequence), KL divergence loss (for regularization), and a frame classification loss (to encourage accurate frame prediction). The authors also explore using different types of FrameNet relations and "scenario" frames to guide the compression layer.

Datasets

The primary dataset used is a subset of the Concretely Annotated Wikipedia dataset. This dataset is derived from English Wikipedia and includes automatically produced FrameNet semantic frame parses, making it suitable for event modeling research. The authors used existing splits for training, validation, and testing, truncating documents to the first 5 events for comparability with prior work.

Key Insights and Ideas

  • Jointly leveraging structural and semantic hierarchy is crucial: The paper demonstrates that combining the structural modeling of event sequences with semantic knowledge from ontologies leads to significant improvements over models that focus on only one aspect.
  • Robustness to missing semantic data: The model's ability to perform well even when a significant portion of semantic frames are masked during training highlights its practical applicability in real-world scenarios where annotations are incomplete.
  • FrameNet relations beyond inheritance are powerful: While inheritance relations are useful, incorporating other semantic relations (like causation, compositionality) from FrameNet, especially "scenario" frames, significantly boosts performance.
  • Hierarchical representation aids event similarity: The learned latent frame representations can be effectively used for downstream tasks like event similarity, demonstrating the model's capability to capture rich semantic information in a zero-shot manner.

Key Results

The SHEM model consistently outperforms existing state-of-the-art methods on several key event modeling tasks:

  • Next Event Prediction (Perplexity): The model achieves lower perplexity scores, indicating better prediction of subsequent events. For example, improvements of up to 8.5% are reported.
  • Long-Range Event Prediction (Wikipedia Inverse Narrative Cloze Score - INC): The model shows significant improvements in predicting longer event sequences, with the best performance often achieved with lower levels of semantic supervision.
  • Event Similarity: The model's learned representations surpass a strong baseline (SWCC) on similarity tasks, demonstrating its ability to capture nuanced semantic relationships between events.
  • Handling Missing Events: The model shows improved robustness in regenerating event sequences when semantically relevant events are masked, outperforming baselines in these scenarios.

Limitations and Future Work

  • Information Bottleneck: The authors acknowledge that the information passed to the compression layer can still be a bottleneck.
  • Ontology Coverage: The reliance on FrameNet means the model's performance is limited by the coverage and universality of the ontology, which is historically developed for English.
  • Pretrained Models and Bias: The use of pretrained models/embeddings means the model may inherit biases present in those models.
  • Future Work: The authors suggest exploring summarization, topic modeling, or other downstream tasks using the learned abstraction concept. They also mention the potential for future work to examine how best to encode the semantics of any particular relation for even better performance.

Summary updated

BibTeX
@inproceedings{dipta2023semantically,
  author = {Shubhashis Roy Dipta and Mehdi Rezaee and Francis Ferraro},
  title = {{Semantically-informed Hierarchical Event Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure