Harvesting Event Schemas from Large Language Models

Jialong Tang, Hongyu Lin, Zhuoqun Li, Yaojie Lu, Xianpei Han, Le Sun, Yijie TangCommunications 2023 · DOI · OpenAlex

Introduction

In natural language processing, an event schema serves as a structural blueprint, defining specific event types (e.g., "Marriage") and their associated semantic roles or "slots" (e.g., "Spouse," "Date," "Location"). These schemas are fundamental to event extraction and the construction of comprehensive knowledge bases. However, traditional manual schema creation faces two significant hurdles: the Open-World Challenge, where new event types emerge faster than humans can catalog them, and the Sparsity/Diversity Challenge, where individual texts rarely contain all possible participants of an event.

To address these limitations, researchers have introduced ESHer (Event Schema Harvester), a framework that leverages the vast world knowledge embedded within Large Language Models (LLMs) to automatically induce event schemas across diverse domains and languages.

The ESHer Framework

ESHer operates as a three-stage pipeline designed to "harvest" implicit knowledge from the internal parameters of LLMs (such as BLOOM) and refine it into structured schemas.

The three-stage ESHer pipeline: Text Conceptualization, Schema Structuralization, and Graph-based Aggregation.
  1. Text Conceptualization via In-Context Generation: Using a few-shot prompting approach, the system asks the LLM to generate candidate schemas for a given text. For example, a sentence about a pandemic might prompt the LLM to suggest a "Spread" event type with slots for "disease," "origin," and "target."
  2. Confidence-aware Schema Structuralization: To mitigate LLM hallucinations and inconsistencies, ESHer filters candidate schemas using metrics for Salience (frequency of a slot for a specific type), Reliability (co-occurrence of slots via PageRank), and Consistency (semantic alignment with the source text).
  3. Graph-based Schema Aggregation: Because individual sentences are often incomplete, ESHer constructs a graph of individual schemas. It employs the Louvain algorithm to cluster these nodes, merging synonyms and aggregating dispersed slots into a single, comprehensive "Complete Schema."

Experimental Evaluation and Results

ESHer was evaluated across multiple datasets, including ERE-EN (English entity/event extraction), ChFinAnn (Chinese financial news), and specialized corpora covering the COVID-19 pandemic and daily news.

Quantitative performance of ESHer across ERE-EN and ChFinAnn datasets, highlighting high acceptability and recall.

The framework demonstrated state-of-the-art performance, significantly outperforming existing clustering methods. On the ERE-EN dataset, ESHer achieved an Adjusted Rand Index (ARI) of 56.59, a massive leap from the previous best of 17.69. Human evaluators found that 85.92% of the event types discovered were acceptable. In the specialized Chinese financial domain, ESHer recovered 100% of human-annotated event types and even identified valid new types previously overlooked by experts.

Discussion and Future Work

ESHer proves that LLMs can function as implicit knowledge bases, capable of "stitching together" complete event frames from fragmented observations. While highly effective, the authors note that future work should focus on improving granularity alignment (distinguishing between general and specific slots) and reducing the computational cost of running large-scale LLMs. Integrating a human-in-the-loop stage could further refine these automated schemas for production-grade applications.

Summary updated

BibTeX
@inproceedings{tang2023harvesting,
  author = {Jialong Tang and Hongyu Lin and Zhuoqun Li and Yaojie Lu and Xianpei Han and Le Sun and Yijie Tang},
  title = {{Harvesting Event Schemas from Large Language Models}},
  booktitle = {Communications},
  year = {2023},
  doi = {10.1007/978-981-99-7224-1\_5},
}

Loading related content...

Figure