ActiveAnno: General-Purpose Document-Level Annotation Tool with Active Learning Integration

Max Wiechmann, Seid Muhie Yimam, Chris BiemannNAACL 2021 · DOI

TL;DR

This paper introduces ActiveAnno, a novel, open-source, web-based annotation tool designed for document-level tasks. It addresses the gap in existing tools by offering a flexible, configurable, and user-friendly platform that integrates active learning and machine learning for efficient, high-quality annotation.

What Problem Does This Paper Address?

Manual annotation is a crucial but often tedious and time-consuming step in many research and industry applications, particularly for tasks like content analysis and supervised machine learning. While existing tools excel at span-level annotations (like Named Entity Recognition), there's a significant gap in robust support for document-level annotation. This means annotating entire documents with labels, sentiments, or topics, which is essential for many modern NLP tasks. Furthermore, many existing tools are either specialized, not open-source, or lack features for efficient, collaborative, and ML-integrated workflows. ActiveAnno aims to fill this void by providing a general-purpose, extensible, and user-friendly solution.

Methods

ActiveAnno is built on a modern, responsive web architecture with a backend service (Kotlin/Ktor) and a frontend (React). Key methodological contributions include:

  • Document-Level Annotation Focus: Explicitly designed for annotating entire documents, moving beyond span-level tasks.
  • Configurable Annotation Schema: Supports various annotation definition types (tag sets, boolean, number, text) and allows flexible project setup.
  • Integrated Machine Learning: Features an "annotation generator" concept that can leverage external ML models or custom logic to provide pre-annotations. This significantly speeds up the annotation process.
  • Active Learning Integration: Implements uncertainty sampling to prioritize documents that the ML model is least confident about, optimizing the use of human annotator time.
  • Extensible Architecture: Designed with a RESTful API and webhooks for seamless integration into existing software systems and workflows.
  • User Roles and Workflow Management: Includes roles like annotator, manager, curator, and producer/consumer, supporting collaborative annotation and quality control.
  • Two-Step Annotation Process: Allows for initial annotation creation, followed by a finalization policy (e.g., majority vote, curator review) to ensure quality.

Datasets

The paper mentions using real-world data for experiments:

  • Retail Feedback Data: Approximately 250,000 German textual feedback comments from a retail context were used to test the system's suitability for industry requirements and evaluate the impact of pre-annotations and active learning.
  • OffensEval 2019 Shared Task (Sub-task A): This publicly available dataset was used to create an example project within ActiveAnno, demonstrating its ability to integrate ML components that can be updated.

No new benchmarks were introduced, but existing datasets were used to showcase ActiveAnno's capabilities.

Key Insights and Ideas

  • Document-Level Annotation is a Critical Gap: The paper strongly emphasizes that document-level annotation is underserved by current tools, and ActiveAnno directly addresses this.
  • ML Integration is Key to Efficiency: By treating ML models as "annotation generators" that provide pre-annotations, ActiveAnno significantly reduces annotator effort and increases throughput.
  • Active Learning Optimizes Human Effort: Uncertainty sampling intelligently guides annotators to the most informative documents, making the annotation process more efficient and effective.
  • Flexibility and Configurability are Paramount: The tool's success hinges on its ability to adapt to diverse annotation tasks and project requirements, which is achieved through its configurable schema and workflow options.
  • Open-Source and Extensible Design: Making the tool open-source (MIT license) and providing robust APIs fosters wider adoption and community contributions.

Key Results

  • Efficiency Gains: In an industry experiment, pre-annotations led to a 28% faster annotation process without compromising annotation accuracy or inter-annotator agreement.
  • Positive User Experience: Annotators reported ActiveAnno as "easy, fast, and convenient to use."
  • Demonstrated ML Integration: The paper showcases successful integration of ML models for pre-annotation and the ability to update these models incrementally.
  • Active Learning Effectiveness: The active learning approach, combined with pre-annotations, further reduced annotator effort by allowing them to select correct options when models achieved acceptable accuracy.

Limitations and Future Work

Limitations:

  • The paper doesn't explicitly detail limitations of the current version, but the focus on document-level annotation implies that span-level tasks might not be as deeply supported as in specialized tools.
  • The effectiveness of the ML integration relies on the quality of external models and their API compliance.

Future Work:

  • In-app Feedback: Implementing direct feedback mechanisms between curators and annotators.
  • More Built-in Generators: Expanding the library of annotation generators for various annotation definition types.
  • UI Improvements: Enhancing the UI for better display of very long documents and adding support for span-level and hybrid (span + document) annotations.
  • Further Extensibility: Continuing to build on the core design principles of extensibility and flexibility.

Summary updated

BibTeX
@inproceedings{wiechmann2021activeanno,
  author = {Max Wiechmann and Seid Muhie Yimam and Chris Biemann},
  title = {{ActiveAnno: General-Purpose Document-Level Annotation Tool with Active Learning Integration}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-demos.12},
}
Figure