SAGEViz: SchemA GEneration and Visualization

Sugam Devare, Mahnaz Koupaee, Gautham Gunapati, Sayontan Ghosh, Sai Vallurupalli, Yash Kumar Lal, Francis Ferraro, Nathanael Chambers, Greg Durrett, Raymond J. Mooney, Katrin Erk, Niranjan BalasubramanianEMNLP 2023 · DOI

TL;DR

This paper introduces SAGEViz, a novel human-in-the-loop system that streamlines the creation and visualization of complex event schemas. By combining the power of Large Language Models (LLMs) with human oversight, SAGEViz significantly reduces the effort required to build high-quality, scalable event representations, making them more accessible for downstream applications.

What Problem Does This Paper Address?

Understanding and reasoning about events is crucial for many AI tasks, from predicting future occurrences to explaining past actions. Event schemas provide a structured way to represent how events unfold in a scenario. However, acquiring these schemas at scale has been a major challenge. Traditional methods of manual annotation are time-consuming and not scalable, while automated methods often produce domain-specific, low-quality schemas that struggle with novel events. This paper addresses the need for an efficient, scalable, and robust method for generating comprehensive event schemas that can handle diverse domains and unseen events.

Methods

SAGEViz employs a two-pronged approach:

  • Schema Curation Component: This is a human-in-the-loop system that leverages LLMs for automated event generation and expansion.
  • Event Generator: Utilizes fine-tuned LLMs (like QGELM) or general-purpose LLMs (like GPT-3.5) to predict subsequent events based on a seed event and context. This can be done in a zero-shot or few-shot manner.
  • Event Enhancement: Human annotators validate and enrich the LLM-generated events by adding details like event-entity relationships, roles, and links to knowledge bases (e.g., Wikidata Q-nodes). This ensures contextual validity and accuracy.
  • Relation Linking: Identifies and validates relationships (hierarchical, temporal, logical) between event pairs to build a structured schema.
  • Entity Coreference Resolution: A crucial step that uses models like SpanBERT to identify and unify mentions of the same entity across different events, enabling more generalized schemas.
  • Schema Visualization Component: A user-friendly interface that allows curators to visualize the evolving schema at various levels of granularity. This helps in understanding the overall structure and drilling down into specific event details.

The system is designed to be modular, allowing for the integration of new LLMs and annotation tools.

Datasets

The paper does not explicitly mention specific datasets used for training the LLMs or for evaluating SAGEViz. However, it refers to previous work that used newswire text and weblog stories for schema induction. The user study involved 35 seeds from 8 domains, suggesting a diverse set of scenarios were used for evaluation.

Key Insights and Ideas

  • Human-AI Collaboration is Key: The paper strongly advocates for a human-in-the-loop approach. LLMs provide the scale and speed for initial generation, while human annotators provide the crucial validation, enrichment, and contextual understanding needed for high-quality schemas.
  • LLMs as Powerful Event Generators: The effectiveness of LLMs like GPT-3.5 in generating relevant and diverse events, even in zero-shot settings, is a significant enabler for automated schema induction.
  • Visualizing Complexity: The ability to visualize schemas at multiple levels of hierarchy is essential for human curators to grasp the overall structure and make informed decisions, especially for complex scenarios.
  • Scalability through Modularity: The plug-and-play nature of the event language models and the overall architecture suggests a path towards building schemas for a wide range of domains without extensive re-engineering.

Key Results

The paper highlights the effectiveness of SAGEViz through a user study. While specific quantitative results are presented in a table, the key findings are:

  • Reduced Interaction Effort: Users using SAGEViz required fewer total steps (7.48 steps on average) to generate schemas compared to a baseline CLI system (12.0 steps).
  • Increased Schema Quality: SAGEViz users were able to select more events (# events: 12.1 vs. 8.8) per interaction.
  • Higher Acceptance Rate: A higher percentage of accepted steps (80.0% vs. 73.3%) indicates that the generated events were more relevant and useful, leading to fewer rejected steps (20.0% vs. 26.7%).
  • Improved Efficiency: The system allows for iterative updates and parallel annotation by multiple users, contributing to more reliable schemas, especially for safety-critical applications.

Limitations and Future Work

  • Manual Role Assignment: Currently, the system relies on manual assignment of semantic roles to entities, which could be automated in the future.
  • LLM Limitations: While LLMs are powerful, they can still produce irrelevant or incorrect events, necessitating human oversight. The paper acknowledges that models built before 2019 struggled with events like COVID-19, highlighting the need for models that can handle unseen world events.
  • Future Work: The authors suggest exploring more advanced LLMs, automating more aspects of the curation process, and further enhancing the visualization capabilities. They also mention the potential for an optional grounding function to link schema elements to external knowledge sources.

Summary updated

BibTeX
@inproceedings{devare2023sageviz,
  author = {Sugam Devare and Mahnaz Koupaee and Gautham Gunapati and Sayontan Ghosh and Sai Vallurupalli and Yash Kumar Lal and Francis Ferraro and Nathanael Chambers and Greg Durrett and Raymond J. Mooney and Katrin Erk and Niranjan Balasubramanian},
  title = {{SAGEViz: SchemA GEneration and Visualization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-demo.29},
}
Figure