SKATE: A Natural Language Interface for Encoding Structured Knowledge

Clifton James McFate, Aditya Kalyanpur, David A. Ferrucci, Andrea Bradshaw, Ariel Diertani, David Melville, Lori MoonAAAI 2021 · DOI

Introduction

Bridging the gap between human language and machine logic remains a significant challenge in artificial intelligence. While humans naturally express complex ideas in fluid prose, machines require rigid, structured data—such as semantic frames or logical formulas—to perform automated reasoning. Standard semantic parsers often struggle with the ambiguity of complex sentences, leading to user frustration. Conversely, manually encoding knowledge into formal logic is a specialized, error-prone task.

SKATE (Structured Knowledge AcquisiTion and Extraction) addresses this "impedance mismatch" by providing a natural language interface that converts informal human language into precise machine-readable knowledge. By employing a neural semantic parser and interactive, semi-structured templates, SKATE enables non-experts to build complex logical rules and policy diagrams without learning formal programming syntax.

System Architecture

The SKATE system is built upon a recursive, template-based architecture designed to guide users through the knowledge encoding process.

The SKATE system architecture, featuring a neural semantic parser and recursive template rendering.

The core components include:

  • Front End UI: Facilitates the "micro-dialogue" between the user and the system.
  • Concept Recognizer: Identifies potential semantic frames and roles within the user's input.
  • SPINDLE Parser: A multi-task neural semantic parser based on the T5 (Text-to-Text Transfer Transformer) model. It simultaneously performs frame disambiguation (identifying word "sense"), span detection (finding relevant sentence parts), and role labeling (assigning semantic roles).
  • Hybrid Fallback: When the neural parser's confidence is low, the system utilizes an unsupervised k-Nearest Neighbor (k-NN) approach with GLoVe embeddings to suggest the most likely frames.
  • Semantic Filtering: A unique feature that uses the selected semantic frame to prune nonsensical autocomplete suggestions, ensuring logical consistency.

Interactive Knowledge Acquisition

SKATE transforms knowledge engineering into an intuitive, form-filling exercise through recursive refinement. Instead of attempting to parse a whole paragraph at once, the system breaks the process into tiny, interactive steps that "coax" the user toward a valid interpretation.

The multi-step interactive process: from initial templates to frame and concept disambiguation.

As a user interacts with the system, they are presented with "fill-in-the-blank" templates. Filling a slot can trigger further templates or prompt the user to disambiguate ambiguous terms. For example, if a user enters a sentence involving the verb "take," the system might ask if they mean "remove an object from a container" or "consume a substance." This interactive approach ensures that the final interpretation is both accurate and machine-executable.

Evaluation and Results

The system was trained and evaluated using several comprehensive datasets, including the Hector Ontology (derived from FrameNet and NOAD) and GLUCOSE (a common-sense knowledge dataset). The SPINDLE parser was trained on 500,000 annotated sentences.

In performance tests, the SPINDLE model achieved 91% accuracy in frame disambiguation and an 84% F1 score for span and role labeling. Human evaluations with domain experts were equally promising: 85% of the entries created via SKATE were rated as perfect paraphrases or minor deviations from the intended complex natural language rules. Experts noted that the tool was significantly faster than manual coding and eliminated syntax-related errors common in formal logic.

Conclusion

SKATE demonstrates the viability of using interactive micro-dialogues to bridge the gap between natural language and structured logic. While current limitations include gaps in the underlying ontology and the need for minimal user training, the system has already shown real-world utility. It was successfully deployed in the PolicyPath mobile app to help organizations formalize complex COVID-19 safety policies. Future work will focus on allowing users to define new frames dynamically and scaling the system through broader crowdsourcing efforts.

Summary updated

BibTeX
@inproceedings{mcfate2021skate,
  author = {Clifton James McFate and Aditya Kalyanpur and David A. Ferrucci and Andrea Bradshaw and Ariel Diertani and David Melville and Lori Moon},
  title = {{SKATE: A Natural Language Interface for Encoding Structured Knowledge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i17.17804},
}
Figure