MIC: A Multi-task Interactive Curation Tool

Shi Yu, Mingfeng Yang, Jerrod Parker, Stephen BrockEMNLP 2022 · DOI

TL;DR

This paper introduces MIC, a novel human-machine collaborative tool designed to streamline multi-task NLP annotation. MIC addresses the high cost and error-proneness of manual annotation by integrating pre-trained models, rules, and dictionaries to provide intelligent suggestions, enabling faster and more accurate labeling across various granularities and types of annotations.

What Problem Does This Paper Address?

High-quality data annotation is a critical bottleneck for the success of many Natural Language Processing (NLP) applications. Acquiring these annotations is often expensive, time-consuming, and prone to human error. While domain adaptation techniques can help fine-tune large NLP models, they struggle with new or niche domains lacking sufficient labeled data, necessitating annotation from scratch. Existing annotation tools often focus on single tasks or lack robust mechanisms for human-machine collaboration, leaving a gap for a flexible, intelligent tool that can accelerate the annotation process across multiple NLP tasks and scales.

Methods

MIC is a web-based annotation system built on Django, Quasar, and Vue.js. Its core innovation lies in its human-machine interactive annotation approach. Key features include:

  • Multi-task and Multi-project Support: MIC can manage diverse annotation tasks across multiple projects and users, allowing for flexible configuration.
  • Intelligent Labeling: It integrates pre-trained ML models, custom rules, and dictionaries to auto-label text, significantly speeding up the process. These models are exposed as RESTFUL APIs, allowing for easy interchangeability.
  • Flexible Annotation Granularity: Users can annotate at different scales, including character/word spans, tokens, lines, or entire documents.
  • Diverse Annotation Types: MIC supports various annotation types, such as sentence-level labels, named entities (NER), and relationship triplets.
  • Collaborative Features: The tool supports multiple users with different roles (e.g., Administrator, Curator, Data Scientist, Reviewer) and includes an adjudication mechanism for resolving conflicts and ensuring annotation quality.
  • Interactive User Interface: The interface is designed for efficient annotation, featuring an automatic labeling zone, an annotation zone, and a summarization zone for reviewing annotations.

Datasets

The paper mentions the use of the following datasets for evaluation:

  • CoNLL2003: A standard benchmark for Named Entity Recognition (NER).
  • NYT Open Relation Extraction Benchmark: A dataset for relation extraction tasks.
  • Proprietary fintech customer support call transcripts: A custom dataset used to obtain three levels of annotations (sentence tagging, NER, and relationship annotation).

Key Insights and Ideas

  • Human-Machine Collaboration is Key: The paper strongly advocates for a collaborative approach where intelligent models assist human annotators, rather than fully automating the process. This hybrid approach leverages the strengths of both.
  • Flexibility is Paramount: MIC's ability to handle multiple annotation types, scales, and projects makes it adaptable to a wide range of real-world NLP challenges, which often involve complex and interconnected annotation needs.
  • Intelligent Suggestions Reduce Cognitive Load: By providing pre-generated labels, MIC allows annotators to focus on correcting errors and handling ambiguous cases, rather than starting from scratch, leading to more efficient work.
  • Summarization for Quality Control: The annotation summarization panel is a crucial feature for reviewers to efficiently assess annotation quality, identify conflicts, and manage contributions from multiple annotators.

Key Results

The paper presents a benchmark study comparing MIC with other annotation tools (Prodigy, YEDDA, GATE) on the CoNLL, NYT, and proprietary call transcript datasets.

  • Reduced Annotation Time: MIC-M (with pre-trained models) significantly reduced annotation time across tasks, with an average time of 38-45 minutes for MIC compared to 63-118 minutes for other tools on the CONNL and NYT tasks.
  • Comparable Accuracy: MIC achieved comparable F1-scores to existing tools, with MIC-M achieving an F1-score of 0.78 on CONNL and 0.59 on NYT, which is competitive with state-of-the-art ML models.
  • Qualitative Benefits: Annotators found MIC particularly helpful for long call transcripts, appreciating its filtering capabilities and smooth switching between annotation levels. They also highlighted the advantage of stacked auto-annotations, especially when pre-trained models were domain-homogeneous.

Limitations and Future Work

  • Limitations: The paper notes that for the NYT task, the MAMA model was slow and produced noisy results, requiring extra effort from annotators and impacting performance. Additionally, pre-trained models were not always helpful for domain-specific entities (e.g., financial products) if they hadn't been adapted to that domain.
  • Future Work: The authors plan to enhance MIC by integrating advanced processes like zero-shot learning, few-shot learning, and textual entailment models. They also aim to implement automated training pipelines for weak supervision algorithms to further improve label denoising.

Summary updated

BibTeX
@inproceedings{yu2022mic,
  author = {Shi Yu and Mingfeng Yang and Jerrod Parker and Stephen Brock},
  title = {{MIC: A Multi-task Interactive Curation Tool}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-demos.22},
}
Figure