Integrating INCEpTION into larger annotation processes

Richard Eckart de Castilho, Jan-Christoph Klie, Iryna Gurevych2024 · DOI

TL;DR

Modern data annotation is no longer a standalone task; it’s a single cog in a complex machine involving machine learning loops, crowdsourcing, and external knowledge bases. This paper outlines the "integrability requirements" for modern annotation tools and demonstrates how INCEpTION, a popular semantic annotation platform, has evolved into a highly extensible "role model" for these integrated workflows.

What Problem Does This Paper Address?

In the early days of NLP, annotation was simple: open a text file, highlight some words, and save the result. Today, the process is far more dynamic. Annotators might need to call a remote LLM for labeling suggestions, pull definitions from a SPARQL endpoint, or work within a massive, shifting team of crowdsourced contributors.

The authors identify a gap between two types of tools:

  1. Instance-oriented tools (like Prodigy or Label Studio): Designed for high-throughput labeling of short snippets but often lacking deep document context.
  2. Document-oriented tools (like brat or WebAnno): Great for complex linguistic analysis but historically rigid and difficult to plug into automated pipelines.

The challenge is making a document-oriented tool that is as "pluggable" and customizable as a modern software API.

Methods

The authors define a framework of five key areas for "integrability":

  • Annotator Management (AM): Handling dynamic teams, self-sign-up (OAuth2/SAML), and reclaiming work from "abandoned" sessions.
  • Task Design (TD): Allowing custom UIs and rendering for complex formats like XML or PDF.
  • Process Integration (PI): Providing robust remote APIs (AERO-compatible) and webhooks to trigger external actions.
  • ML Services (ML): Supporting "predict/fit" paradigms where models provide suggestions and learn from annotator corrections in real-time.
  • Knowledge Bases (KB): Connecting to massive external terminologies via SPARQL or a new, lightweight lookup protocol (LLP).

The paper details how INCEpTION implemented these features, specifically highlighting a new JavaScript API that allows developers to build custom editor plugins (e.g., for side-by-side document comparison) without modifying the core INCEpTION codebase.

Datasets

As this is a System Demonstration paper, it does not introduce a new benchmark dataset. Instead, it demonstrates the tool's flexibility using various formats:

  • TEI P5 XML: Used to show custom rendering of plays from the Drama Corpora Project.
  • PDF and HTML: Demonstrating layout-oriented annotation.
  • Cross-document linking: Showcased via the INTERTEXT plugin for pairwise document analysis.

Key Insights and Ideas

  • The "Abandoned Work" Problem: In document-oriented tools, an annotator might spend 30 minutes on one file and then leave. INCEpTION introduced specific logic to "reclaim" this work—either discarding it, locking it for the original user, or marking it "finished" despite being incomplete—to prevent pipeline bottlenecks.
  • Decoupling the UI: By introducing a JavaScript API, the authors moved away from a "one-size-fits-all" editor. This allows for specialized tasks like word alignment or cross-document linking to be built as plugins that talk to the backend via a standardized protocol.
  • Lightweight Lookup Protocol (LLP): While SPARQL is powerful, it’s often overkill. The authors introduced a simpler HTTP-based protocol to make it easier for developers to connect their own custom databases or search engines (like Solr or Elasticsearch) to the annotation interface.

Key Results

The paper provides a comprehensive comparison (Table 1) of INCEpTION against 10 other popular tools (including Doccano, Prodigy, and Label Studio).

  • Versatility: INCEpTION is shown to be one of the few tools supporting nearly all identified integrability requirements, particularly in document-level management and knowledge base integration.
  • Extensibility: The system now supports custom XML styling via CSS and a "predict/fit" paradigm for active learning, which was previously difficult to implement in document-heavy environments.
  • Enterprise Readiness: Support for OAuth2 and SAML-compliant identity providers (introduced around v0.27.0) makes it viable for large-scale institutional use.

Limitations and Future Work

The authors acknowledge that their survey is limited to a selection of popular and recent tools and may have overlooked some niche software.

Looking ahead, they see a major opportunity in innovative UIs for Large Language Models (LLMs). As the role of the annotator shifts from "creating labels" to "evaluating and correcting LLM outputs," the interface needs to evolve to support these specific interaction patterns. They also aim to continue expanding the range of supported document types and specialized annotation tasks.

Summary updated

Abstract

Annotation tools are increasingly only steps in a larger process into which they need to be integrated, for instance by calling out to web services for labeling support or importing documents from external sources.This requires certain capabilities that annotation tools need to support in order to keep up.Here, we define the respective requirements and how popular annotation tools support them.As a demonstration for how these can be implemented, we adapted INCEpTION, a semantic annotation platform offering intelligent assistance and knowledge management.For instance, support for a range of APIs has been added to INCEp-TION through which it can be controlled and which allow it to interact with external services such as authorization services, crowdsourcing platforms, terminology services or machine learning services.Additionally, we introduce new capabilities that allow custom rendering of XML documents and even the ability to add new JavaScript-based editor plugins, thereby making INCEpTION usable in an even wider range of annotation tasks.
BibTeX
@misc{castilho2024integrating,
  author = {Richard Eckart de Castilho and Jan-Christoph Klie and Iryna Gurevych},
  title = {{Integrating INCEpTION into larger annotation processes}},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-demo.12},
}
Figure