A Graphical Interface for Curating Schemas

Piyush Mishra, Akanksha Malhotra, Susan Windisch Brown, Martha Palmer, Ghazaleh KazeminejadACL 2021

TL;DR

This paper introduces a web-based graphical interface designed to help humans visualize and edit complex event schemas automatically generated by AI systems. By transforming abstract JSON data into an intuitive visual representation, the tool aims to bridge the gap between machine-generated insights and human understanding, facilitating more accurate and nuanced schema curation.

What Problem Does This Paper Address?

Understanding complex sequences of events, like emergency medical responses or improvised explosive device (IED) scenarios, is crucial for many AI applications, including information extraction and knowledge base construction. While AI systems can now generate these complex "schemas" (structured representations of events, participants, and their relationships), these machine-generated outputs are often difficult for humans to read, review, and refine due to their abstract, linear format (typically JSON). This makes it challenging to correct errors, add nuance, or ensure the schemas accurately reflect real-world scenarios. The paper addresses the need for a user-friendly tool that allows for the effective visualization and manual curation of these complex, machine-induced event schemas.

Methods

The core of the paper is the description of the "Schema Curation Interface," a web application built using React.js for the frontend and Flask for the backend. The interface takes schemas in a JSON-LD format (specifically, the KAIROS Schema Format or KSF) as input. It then uses Cytoscape.js, a JavaScript graph library, to render these schemas as visual graphs on a canvas.

Key methodological aspects include:

  • Dual-Layer Visualization: The interface offers two views: a default view showing only events and their temporal relations, and a second view that reveals entities and their roles within specific events. This helps reduce visual clutter.
  • Interactive Editing: Users can directly edit the underlying JSON schema through a linked JSON viewer. Changes made in the JSON are dynamically reflected in the graphical visualization, and vice-versa.
  • Node and Edge Representation: Events, entities (participants/slots), and fillers (values) are represented as distinct nodes with different shapes (e.g., ellipses for events, peach pentagons for entities). Edges represent relationships, with different edge types indicating temporal or causal connections.
  • Temporal Progression: The left-to-right layout of the graph visually represents temporal progression.
  • Containerization: The application is deployed using Docker and Kubernetes on Google Cloud Platform (GCP) for scalability and reliability.

Datasets

The paper does not explicitly mention specific datasets used for training the underlying Information Extraction (IE) systems that generate the initial schemas. However, it discusses the use of "complex event schemas" derived from "large amounts of text" and "streaming news feeds" as part of the DARPA KAIROS program. The authors also mention using "wiki articles about various IED attacks" to inform the curation process and determine temporal ordering. The schemas themselves, in the KSF format, are the primary data handled by the interface.

Key Insights and Ideas

  • Bridging the Human-AI Gap: The most significant insight is the critical need for human-in-the-loop curation for complex AI-generated knowledge structures. While AI can induce schemas, human expertise is essential for validation, refinement, and ensuring accuracy.
  • Visualization as a Tool for Understanding: The paper strongly emphasizes how graphical visualization dramatically improves the understanding of complex, multi-layered event schemas, which are otherwise opaque in their raw JSON format. This is particularly true for identifying parallel events and tracking participants across multiple subevents.
  • Iterative Refinement: The process of schema curation is presented as iterative. The interface facilitates multiple rounds of editing, comparison, and re-curation, leading to progressively improved and enriched schemas.
  • Flexibility in Schema Representation: The interface supports the representation of various schema complexities, including linear event chains and more intricate structures with parallel and optional events, as well as temporal and causal relations.

Key Results

The paper focuses on the development and utility of the interface rather than presenting quantitative performance metrics of the schema induction systems themselves. The key results are qualitative:

  • Improved Schema Quality: The interface enabled human curators to identify and correct errors, add missing events, prune unnecessary ones, and refine temporal and causal relationships, resulting in "improved and enriched versions" of machine-induced schemas.
  • Facilitated Collaboration: The tool was used by approximately 10 people across three institutions, demonstrating its utility for collaborative schema curation.
  • Enhanced Understanding of Complex Events: The visualization allowed curators to more easily understand the structure of complex schemas, including those related to IED attacks, medical interventions, and general attack scenarios.
  • Efficient Editing: The ability to edit directly in JSON and see immediate visual updates, or vice-versa, streamlined the curation process.

Limitations and Future Work

Limitations:

  • Hierarchical Schemas: The current implementation of the KAIROS Schema Format (KSF) restricts hierarchical schemas, meaning subschemas cannot be collapsed into a single parent node. This can lead to lengthy and potentially cluttered visualizations.
  • Schema Size: While the dual-layer view helps, very complex schemas with many events can still be difficult to visualize legibly, sometimes requiring manual arrangement of nodes.
  • Limited Primitive Events: The current set of primitive events is small, which limits the scope of complex events that can be fully represented.

Future Work:

  • Hierarchical Schema Handling: The authors plan to improve the handling of hierarchical schemas, allowing for more compact and organized representations.
  • Dynamic Visualization: Enhancing the dynamism of the visualization is a goal, potentially allowing for more interactive exploration.
  • Schema Comparison: Developing better tools for comparing different versions of schemas or schemas from different sources.
  • Direct Graph Editing: Expanding direct editing capabilities to the graph visualization itself, beyond just the JSON editor.
  • More Primitive Events: Incorporating a wider range of primitive events to increase the expressiveness of the schemas.
  • Alternative Instigating Events: Adding more types of instigating events for scenarios like attacks.

Summary updated

BibTeX
@inproceedings{mishra2021graphical,
  author = {Piyush Mishra and Akanksha Malhotra and Susan Windisch Brown and Martha Palmer and Ghazaleh Kazeminejad},
  title = {{A Graphical Interface for Curating Schemas}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure