DoTAT: A Domain-oriented Text Annotation Tool

Yupian Lin, Tong Ruan, Ming Liang, Tingting Cai, Du Wen, Yi Wang2022 · DOI

TL;DR

DoTAT is a new web-based annotation tool specifically designed for the "messy" reality of domain-specific NLP, such as medical or legal texts. It streamlines the creation of high-quality datasets by supporting complex nested events and entities, while offering a clever multi-person collaborative workflow that uses the Kuhn-Munkres algorithm to automatically merge and flag inconsistent annotations for review.

What Problem Does This Paper Address?

In the world of Natural Language Processing (NLP), high-quality labeled data is the "gold" that powers machine learning models. However, creating this data is notoriously slow, expensive, and prone to human error.

The authors identify three major gaps in existing tools like Brat or WebAnno:

  1. Complexity: Domain-specific tasks often require "nested" structures (e.g., a medical "treatment" event that contains a "dosage" entity, which itself might be part of a larger "patient history" event). Most tools struggle to visualize these without becoming a tangled mess of overlapping lines.
  2. Collaboration Friction: Single-person annotation is unreliable. While multi-person workflows are better, merging different people's work and resolving their disagreements is usually a manual, grueling process.
  3. Rigid Specifications: Domain requirements change frequently. Most tools make it difficult to update annotation schemas or re-use old work when the rules change.

Methods

DoTAT (Domain-oriented Text Annotation Tool) introduces several features to tackle these issues:

  • Panel-Based Visualization: Instead of the "spaghetti" of arcs seen in older tools, DoTAT uses a dual-panel system. Users select text in an annotation panel, and the relationships are summarized in a clean event list panel. For nested entities, they use a "layered shadow" approach to make overlapping labels visible.
  • Automated Batch Annotation: To jumpstart the process, users can define regular expressions or dictionaries. The tool then pre-annotates the text, leaving the human to simply verify or refine the results.
  • Smart Merging (The Kuhn-Munkres Algorithm): This is the "secret sauce." When two people annotate the same text, DoTAT treats merging as a "maximum weight matching" problem in a bipartite graph. It calculates similarity scores between annotations and automatically categorizes them into four states: Consistent, Only A, Only B, or Inconsistent.
  • Iterative Workflow: The tool supports a "Define → Assign → Annotate → Merge/Review → Export" loop, allowing teams to refine their specifications and re-import data as the project evolves.

Datasets

The researchers evaluated DoTAT using:

  • ACE2005: A classic benchmark dataset for entity and event extraction.
  • Public Security Domain: A custom dataset of 10,000 texts related to criminal fraud cases.
  • Medical Domain: A dataset of 300 long-form medical texts.

Key Insights and Ideas

  • Reviewers are the Bottleneck: The authors realized that the reviewer's time is the most precious. By using an algorithm to automatically merge "Consistent" annotations, the reviewer can focus exclusively on the "Inconsistent" cases where humans actually disagreed.
  • UI Matters for Accuracy: DoTAT achieved higher accuracy than Brat even before the review phase. The authors attribute this to the UI: when a user selects a text span, a pop-up menu reminds them of all required arguments for that event type, making it much harder to accidentally skip a field.
  • Nested is the New Normal: In specialized fields, entities rarely exist in isolation. Designing for nesting from the ground up—rather than as an afterthought—is essential for modern information extraction.

Key Results

The experimental results on the ACE2005 dataset were impressive:

  • Speedup: DoTAT reduced the time required for event annotation by 19.7% compared to existing tools.
  • Higher Baseline Accuracy: Without any review, DoTAT achieved 84.09% accuracy, outperforming Brat (82.74%) and WebAnno (81.5%).
  • Gold Standard Quality: After the automated merging and human review process, the final accuracy reached 93.76%.
  • Real-World Scale: In the public security case study, the tool successfully managed a massive project involving 80,000 entities and 20,000 events.

Limitations and Future Work

While DoTAT significantly speeds up the process, it still relies heavily on manual labor. The authors note that even with batch annotation, humans are doing the heavy lifting.

Future Work: The team plans to integrate active learning algorithms. This would allow the tool to "learn" as the humans annotate, eventually identifying which specific sentences are the most difficult or informative, and presenting those to the annotators first to maximize the model's learning rate.

Reference: The code for DoTAT is open-source and available on GitHub at https://github.com/FXLP/MarkTool.

Summary updated

Abstract

We propose DoTAT, a domain-oriented text annotation tool. The tool designs and implements functions heavily in need in domain-oriented information extraction. Firstly, the tool supports a multi-person collaborative process with automatically merging and review, which can greatly improve the annotation accuracy. Secondly, the tool provides annotation of events, nested event and nested entity, which are frequently required in domain-related text structuring tasks. Finally, DoTAT provides visual annotation specification definition, automatic batch annotation and iterative annotation to improve annotation efficiency. Experiments on the ACE2005 dataset show that DoTAT can reduce the event annotation time by 19.7% compared with existing annotation tools. The accuracy without review is 84.09%, 1.35% higher than Brat and 2.59% higher than Webanno. The accuracy of DoTAT even reaches 93.76% with review.
BibTeX
@misc{lin2022dotat,
  author = {Yupian Lin and Tong Ruan and Ming Liang and Tingting Cai and Du Wen and Yi Wang},
  title = {{DoTAT: A Domain-oriented Text Annotation Tool}},
  year = {2022},
  doi = {10.18653/v1/2022.acl-demo.1},
}
Figure