A Human-machine Interface for Few-shot Rule Synthesis for Information Extraction

Robert Vacareanu, George C. G. Barbosa, Enrique Noriega-Atala, Gus Hahn-Powell, Rebecca Sharp, Marco A. Valenzuela-Escárcega, Mihai SurdeanuNAACL 2022

TL;DR

This paper introduces a human-machine interface that allows users to synthesize information extraction rules using natural language examples, even with minimal NLP expertise. The system leverages a pre-trained neural synthesizer to generate high-precision rules in a few-shot setting, outperforming manually written patterns on a standard dataset.

What Problem Does This Paper Address?

Traditional rule-based systems for information extraction offer transparency and interpretability, which are often lacking in deep learning models. However, these systems typically require users to be proficient in declarative rule languages and complex linguistic representations, limiting their adoption by domain experts outside of NLP. This paper addresses the challenge of making rule-based information extraction accessible to a broader audience by developing a system that bridges the gap between natural language examples and formal extraction rules.

Methods

The core of the system is a rule synthesis engine that combines an enumerative searcher with a transformer-based neural scorer. The searcher explores potential rule candidates, guided by the scorer, which prioritizes promising paths. This scorer is pre-trained on a large corpus of automatically generated rules and then fine-tuned using self-supervision. The system operates in a few-shot manner, meaning it can generate rules from a very small number of examples provided by the user. The user interacts with the system through a web-based interface where they can provide examples, review proposed rules and their extractions, and refine their input. The system supports both positive and negative examples to guide the rule generation process.

Datasets

  • UMBC corpus: Used for generating training data for the neural scorer and for evaluating the interactive sessions.
  • TACRED dataset: A standard relation classification dataset used for a non-interactive evaluation of the synthesized rules.

Key Insights and Ideas

  • Democratizing Rule Synthesis: The most significant insight is that users without deep NLP knowledge can effectively generate high-precision information extraction rules through an intuitive example-driven interface.
  • Few-Shot Learning for Rules: The system's ability to synthesize rules from a single example (1-shot setting) is a crucial advancement, significantly lowering the barrier to entry for creating custom extraction models.
  • Pre-trained Synthesizer as a Prompt: The approach of pre-training a general-purpose rule synthesizer and then "prompting" it with specific examples is analogous to how large language models are used, demonstrating a powerful paradigm for task adaptation.
  • Human-in-the-Loop Efficiency: The interactive evaluation highlights that users can achieve high precision with only a few iterations, suggesting the system is efficient for domain experts needing quick, interpretable extraction solutions.

Key Results

  • Interactive Evaluation: In a 1-shot setting, the system generated rules with high precision, achieving P@20 of over 75% for the CITY entity type. This was demonstrated across various entity types, including coarse-grained ones like CITY and PERSON, and finer-grained ones like CAPITAL and ACTOR.
  • Non-interactive Evaluation (TACRED): The synthesized rules outperformed manually written patterns on the TACRED dataset by over 4 F1 points, even though the synthesis component was not specifically trained on TACRED data. The contextualized weights model achieved an F1 score of 41.4, outperforming seq2seq baselines (28.0 F1) and a rule-based baseline (36.6 F1).

Limitations and Future Work

  • "Surface" Rules: The current rules operate on sequences of tokens. Future work aims to support rules that can capture more complex relationships by incorporating syntactic information and handling longer-distance dependencies.
  • Single Rule Generation: The system currently generates a single rule to capture the entire specification. The authors suggest that generating multiple simpler rules could be a more effective approach.
  • User Study on Preferences: The system tends to propose overly general rules initially. Further user studies are needed to understand user preferences for precision vs. recall in early iterations and to potentially bias the system's output accordingly.

Summary updated

BibTeX
@inproceedings{vacareanu2022human,
  author = {Robert Vacareanu and George C. G. Barbosa and Enrique Noriega-Atala and Gus Hahn-Powell and Rebecca Sharp and Marco A. Valenzuela-Escárcega and Mihai Surdeanu},
  title = {{A Human-machine Interface for Few-shot Rule Synthesis for Information Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}

Loading related content...

Figure