Mining Logical Event Schemas From Pre-Trained Language Models

Lane Lawley, Lenhart SchubertACL 2022

TL;DR

This paper introduces NESL, a novel system that leverages large pre-trained language models (LLMs) to automatically learn structured, logical representations of event schemas. By creatively prompting LLMs to generate "situation samples" and then applying a pipeline of semantic parsing and schema generalization, NESL bridges the gap between rich symbolic representations and the flexibility of neural models, enabling more comprehensive and controllable schema learning.

What Problem Does This Paper Address?

The acquisition of event schemas – structured representations of how events unfold and relate to each other – has historically been a challenge. Traditional approaches fall into two camps: symbolic methods, which offer rich, interpretable representations but are brittle and difficult to scale, and statistical methods, which are more flexible but often lack representational depth and control over what is learned. This paper aims to bridge this divide by using LLMs to generate diverse "situation samples" and then employing a hybrid neuro-symbolic pipeline to extract and generalize these into rich, logical event schemas. This approach addresses the need for scalable, yet semantically robust, event schema learning that can be guided by user prompts.

Methods

NESL employs a multi-step pipeline:

  1. Latent Schema Sampling (LSS): This is the core innovation. Instead of relying on a pre-defined corpus, NESL uses a pre-trained LLM (GPT-J 6B) to generate "situation samples." This is achieved through "prompt-engineering," where the LLM is guided to act as a distribution over stories parameterized by an implicit latent schema. By sampling multiple stories from this distribution and identifying commonalities while discarding infrequent details, NESL aims to extract more accurate and less noisy schemas.
  2. Topic Extraction: A few-shot prompt is used to extract 1-3 simple topics from a curated set of ROCStories to guide the LLM.
  3. Story Sampling: For each topic, the LLM generates multiple stories (N) using a specified temperature and repetition penalty, filtered for inappropriate content.
  4. Semantic Parsing: The generated stories are parsed into Episodic Logic (EL), a formal, intensional semantic representation that treats events as first-class individuals and supports complex temporal relations.
  5. Protoschema Matching: A neural FrameNet parser (LOME) identifies semantic frames in the stories. These frames are then mapped to a set of pre-defined, simple "protoschemas" (e.g., "X helps Y with action A").
  6. Single-Story Schema Formation: For each sampled story, the parsed EL and matched protoschemas are combined into a single-story schema. This involves abstracting individuals to variables, normalizing verbiage, and creating a temporal graph.
  7. Multi-Story Schema Generalization: The N single-story schemas are then generalized into a single, comprehensive schema. This process involves:
  8. Schema Step Clustering: Using vector embeddings of EL formulas to group similar steps across stories.
  9. Step Argument Co-reference Resolution: Constructing a multigraph to resolve argument co-references between clustered steps.
  10. Temporal Sorting: Ordering the general schema's steps using a temporal multi-graph.
  11. Occurrence Frequency Filtering: Removing infrequent steps that are likely noise.
  12. Verbalization and Rendering: The final EL schemas are verbalized into human-readable English using GPT-2 and rendered in HTML.

Datasets

  • ROCStories: Used to filter for stories with a high proportion of words shared with classic children's stories, creating a corpus for topic extraction.
  • Pre-trained Language Models: GPT-J 6B for story generation and GPT-2 for verbalization.
  • FrameNet: Used via the LOME parser to identify semantic frames.
  • Manual Annotations: 1,200 pairs of pseudo-English EL representations and their English translations were used to fine-tune the GPT-2 verbalization model.

No new benchmarks were introduced, but the system leverages existing datasets and models.

Key Insights and Ideas

  • Latent Schema Sampling (LSS): The core idea that LLMs can be prompted to implicitly encode latent schemas, and that sampling from this distribution can yield more robust and less noisy data for schema learning. This bypasses the need for curated, large-scale narrative datasets.
  • Hybrid Neuro-Symbolic Approach: Effectively combining the strengths of LLMs (flexibility, knowledge) with formal semantic representations (richness, interpretability) and structured schema learning.
  • Protoschemas as Building Blocks: Using simple, universal protoschemas, inspired by Piaget's developmental theories, as a foundation for learning more complex hierarchical schemas.
  • Controllable Schema Generation: LSS allows for user-guided schema generation by specifying topics or prompts, offering more control than purely unsupervised methods.

Key Results

The paper demonstrates that NESL can generate general, semantically rich, and interpretable event schemas from LLM-generated stories. The authors show examples of learned schemas for prompts like "going fishing," "going to the library," and "mailing a letter." While quantitative evaluation results are not presented in this paper (as it's a work in progress), the qualitative results showcase the system's ability to capture nuanced event sequences and participant roles. The system can generate a schema in approximately 10 minutes.

Limitations and Future Work

  • Qualitative Evaluation: The primary limitation acknowledged is the lack of comprehensive qualitative evaluation of the generated schemas.
  • Future Evaluation: The authors plan to conduct human-judged studies to evaluate:
  • The quality of inferences generated by the learned schemas on unseen stories.
  • The overall quality of the schemas themselves.
  • Inferential Capacity: Demonstrating the inferential capabilities of the learned schemas is a key future direction, for which they will source suitable inference datasets.
  • Schema Cohesion and Interest: Evaluating whether schemas are topically cohesive and capable of generating interesting, non-obvious inferences.
  • Temporal Model Extension: The current temporal model assumes only start times; extending it to handle both start and end times is a theoretical possibility.
  • Certainty-Weighted Disjunctions: The paper notes that displaying certainty-weighted disjunctions for variables with multiple possible types is intended for a future version.

Summary updated

BibTeX
@inproceedings{lawley2022mining,
  author = {Lane Lawley and Lenhart Schubert},
  title = {{Mining Logical Event Schemas From Pre-Trained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure