TL;DR
Modern data annotation is no longer a standalone task; it’s a single cog in a complex machine involving machine learning loops, crowdsourcing, and external knowledge bases. This paper outlines the "integrability requirements" for modern annotation tools and demonstrates how INCEpTION, a popular semantic annotation platform, has evolved into a highly extensible "role model" for these integrated workflows.
What Problem Does This Paper Address?
In the early days of NLP, annotation was simple: open a text file, highlight some words, and save the result. Today, the process is far more dynamic. Annotators might need to call a remote LLM for labeling suggestions, pull definitions from a SPARQL endpoint, or work within a massive, shifting team of crowdsourced contributors.
The authors identify a gap between two types of tools:
- Instance-oriented tools (like Prodigy or Label Studio): Designed for high-throughput labeling of short snippets but often lacking deep document context.
- Document-oriented tools (like brat or WebAnno): Great for complex linguistic analysis but historically rigid and difficult to plug into automated pipelines.
The challenge is making a document-oriented tool that is as "pluggable" and customizable as a modern software API.
Methods
The authors define a framework of five key areas for "integrability":
- Annotator Management (AM): Handling dynamic teams, self-sign-up (OAuth2/SAML), and reclaiming work from "abandoned" sessions.
- Task Design (TD): Allowing custom UIs and rendering for complex formats like XML or PDF.
- Process Integration (PI): Providing robust remote APIs (AERO-compatible) and webhooks to trigger external actions.
- ML Services (ML): Supporting "predict/fit" paradigms where models provide suggestions and learn from annotator corrections in real-time.
- Knowledge Bases (KB): Connecting to massive external terminologies via SPARQL or a new, lightweight lookup protocol (LLP).
The paper details how INCEpTION implemented these features, specifically highlighting a new JavaScript API that allows developers to build custom editor plugins (e.g., for side-by-side document comparison) without modifying the core INCEpTION codebase.
Datasets
As this is a System Demonstration paper, it does not introduce a new benchmark dataset. Instead, it demonstrates the tool's flexibility using various formats:
- TEI P5 XML: Used to show custom rendering of plays from the Drama Corpora Project.
- PDF and HTML: Demonstrating layout-oriented annotation.
- Cross-document linking: Showcased via the INTERTEXT plugin for pairwise document analysis.
Key Insights and Ideas
- The "Abandoned Work" Problem: In document-oriented tools, an annotator might spend 30 minutes on one file and then leave. INCEpTION introduced specific logic to "reclaim" this work—either discarding it, locking it for the original user, or marking it "finished" despite being incomplete—to prevent pipeline bottlenecks.
- Decoupling the UI: By introducing a JavaScript API, the authors moved away from a "one-size-fits-all" editor. This allows for specialized tasks like word alignment or cross-document linking to be built as plugins that talk to the backend via a standardized protocol.
- Lightweight Lookup Protocol (LLP): While SPARQL is powerful, it’s often overkill. The authors introduced a simpler HTTP-based protocol to make it easier for developers to connect their own custom databases or search engines (like Solr or Elasticsearch) to the annotation interface.
Key Results
The paper provides a comprehensive comparison (Table 1) of INCEpTION against 10 other popular tools (including Doccano, Prodigy, and Label Studio).
- Versatility: INCEpTION is shown to be one of the few tools supporting nearly all identified integrability requirements, particularly in document-level management and knowledge base integration.
- Extensibility: The system now supports custom XML styling via CSS and a "predict/fit" paradigm for active learning, which was previously difficult to implement in document-heavy environments.
- Enterprise Readiness: Support for OAuth2 and SAML-compliant identity providers (introduced around v0.27.0) makes it viable for large-scale institutional use.
Limitations and Future Work
The authors acknowledge that their survey is limited to a selection of popular and recent tools and may have overlooked some niche software.
Looking ahead, they see a major opportunity in innovative UIs for Large Language Models (LLMs). As the role of the annotator shifts from "creating labels" to "evaluating and correcting LLM outputs," the interface needs to evolve to support these specific interaction patterns. They also aim to continue expanding the range of supported document types and specialized annotation tasks.
Summary updated