DERE: A Task and Domain-Independent Slot Filling Framework for Declarative Relation Extraction

Heike Adel, Laura Ana Maria Bostan, Sean Papay, Sebastian Padó, Roman KlingerEMNLP 2018

TL;DR

This paper introduces DERE, a novel framework for declarative relation extraction that aims to bridge the gap between task-specific NLP models and the need for generalizability and reusability. By decoupling task specification from model implementation, DERE allows researchers and developers to easily define, train, and evaluate models across diverse information extraction tasks without extensive re-engineering.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) is characterized by a proliferation of specialized models, each tailored to a specific task like relation extraction, semantic role labeling, or event extraction. This specialization leads to several challenges:

  • Lack of Generalizability: Lessons learned from one task are difficult to transfer to another.
  • Lack of Comparability: It's hard to isolate the true advancements in core models when end-to-end evaluations include peripheral components.
  • Difficulty of Reusability: Reusing specific code or models within complex systems is cumbersome.
  • Difficulty of Usage: End-users without deep machine learning expertise struggle to utilize or adapt existing IE tools.

These limitations hinder progress by making it difficult to compare models, reuse existing work, and apply NLP tools to new problems efficiently.

Methods

DERE tackles these challenges through a two-pronged approach:

  1. Declarative Task Specification (Frontend): DERE uses a generic, XML-based "task schema" to define information extraction tasks. This schema builds upon two fundamental concepts:
  2. Spans: Contiguous segments of text representing entities or triggers.
  3. Frames: Representing n-ary relations, where each frame is anchored by a "trigger" span and contains typed "slots" that can be filled by other spans or frames. This formalism is designed to be flexible enough to represent a wide range of IE tasks.

  4. Modular Backend (Backend): The framework separates the task specification from the actual prediction models. This "decoupling" allows for interchangeable backends that can implement various modeling paradigms (e.g., traditional machine learning, deep learning). The backend receives the task schema and annotated data to train and run models.

The paper also presents a proof-of-concept system that uses a pipeline approach:

  • Span Identification: Identifies spans relevant to the task, using Conditional Random Fields (CRFs) with task-agnostic features.
  • Slot Classification: Predicts which slots of which frames spans are likely to fill, using Support Vector Machines (SVMs) on span pairs.
  • Decoding: Compiles the classified slots into frames, ensuring consistency with the task schema.

Datasets

The paper uses the following datasets for evaluation:

  • BioNLP'09 Shared Task: Used for evaluating the proof-of-concept system on biomedical relation extraction (specifically, event extraction).
  • USAGE Corpus: Used for evaluating the proof-of-concept system on aspect-based sentiment analysis.

The paper does not introduce new datasets but rather demonstrates the framework's ability to handle existing benchmarks by adapting task schemas.

Key Insights and Ideas

  • Task-Domain Independence is Achievable: The core idea is that by abstracting away task-specific details into a declarative schema, the underlying models can be made more general and reusable.
  • Spans and Frames as a Universal Language: The framework's reliance on spans and frames provides a unified way to represent diverse IE tasks, from simple entity recognition to complex event structures.
  • Decoupling Frontend and Backend is Crucial: Separating the "what" (task specification) from the "how" (model implementation) is key to enabling easy model swapping, cross-task evaluation, and transfer learning.
  • Simplified User Experience: DERE aims to lower the barrier to entry for users without deep ML expertise, allowing them to define and use IE systems more intuitively.

Key Results

The paper reports on the performance of its proof-of-concept system on two tasks:

  • BioNLP'09 Shared Task: The system achieved competitive results, performing slightly lower than the second-ranked system but demonstrating the general usability of the framework. For example, it achieved an F1 score of 57.97% for SVT-TOTAL.
  • Aspect-Based Sentiment Analysis (USAGE Corpus): The results were noted as "very low" compared to previous work, which the authors attribute to the lack of joint inference in their proof-of-concept system. However, they highlight that the framework successfully allowed them to apply the same model by simply changing the task schema, showcasing its reusability.

Limitations and Future Work

Limitations:

  • The proof-of-concept system is limited to non-recursive structures, meaning slots cannot be filled by other frames.
  • The BRAT annotation format, while supported, is not as expressive as the DERE task schema.

Future Work:

  • Develop joint models that can handle recursive structures and more complex dependencies.
  • Integrate state-of-the-art neural network models into the backend.
  • Support a native, XML-based annotation format for richer task specifications.
  • Address research questions related to transfer learning and joint learning across tasks and domains.
  • Organize a multi-task oriented shared task to further explore the integration of different model types and configurations.

Summary updated

BibTeX
@inproceedings{adel2018dere,
  author = {Heike Adel and Laura Ana Maria Bostan and Sean Papay and Sebastian Padó and Roman Klinger},
  title = {{DERE: A Task and Domain-Independent Slot Filling Framework for Declarative Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure