Anafora: A Web-based General Purpose Annotation Tool

Wei-Te Chen, Will StylerNAACL 2013

TL;DR

Anafora is a new, open-source, web-based tool designed for flexible and efficient text annotation. It addresses the limitations of traditional local annotation tools by offering a centralized, browser-accessible platform that supports complex annotation schemas and workflows, making it suitable for both small and large-scale projects.

What Problem Does This Paper Address?

Annotating text data for natural language processing tasks is a crucial but often cumbersome process. Existing tools frequently require local installation, leading to cross-platform deployment issues and complications with data-use agreements, especially for sensitive data like medical records. Furthermore, many tools struggle with supporting complex annotation schemas, which involve intricate relationships between annotated entities and their properties. This gap creates a need for a more accessible, flexible, and robust annotation solution that can handle diverse and sophisticated annotation requirements.

Methods

Anafora is a web-based application built using Django on the server-side and jQuery on the client-side. Its core design principles emphasize:

  • Web-Based Architecture: This allows for easy deployment, updates, and remote access from any modern OS without local installation or storage requirements.
  • Lightweight and Flexible Schema: Anafora uses human-readable XML for both annotation schemas and data storage. This simplifies schema creation and modification, allowing for complex, multi-slot relations and properties.
  • Centralized File Storage: Data is stored in a simple, static filesystem, eliminating the need for complex databases and simplifying administration and data management.
  • Automatic Set Assignment and Progress Tracking: The tool automates task assignment for annotators and provides progress tracking for administrators.
  • Support for Spanned and Relation Annotations: Anafora handles both entity (spanned) and relation annotations, including adjudication for both.
  • Keyboard and Mouse Interaction: The interface supports both input methods for improved efficiency.

Datasets

Not discussed in the paper. The paper focuses on the tool itself and its capabilities rather than specific datasets used for training or evaluation.

Key Insights and Ideas

  • Web-Based Annotation as a Solution to Deployment and Data Sharing Issues: By moving annotation to a web-based platform, Anafora sidesteps common problems associated with local installations, such as cross-platform compatibility and the complexities of sharing sensitive data.
  • XML-Based Schemas for Enhanced Flexibility: The use of human-readable XML for defining annotation schemas allows for a high degree of customization, supporting complex annotation types and properties that are difficult to manage with more rigid systems.
  • Centralized, File-Based Storage Simplifies Administration: Eliminating the need for a database and relying on a static filesystem makes project setup, data management, and backups significantly easier.
  • Designed for Complex Workflows: Anafora is explicitly built to handle sophisticated annotation tasks, such as those found in medical named entity recognition or coreference resolution, which often involve multi-step processes and intricate relationships.

Key Results

The paper primarily describes the features and design of Anafora, highlighting its advantages over existing tools. While specific quantitative results are not presented, the key qualitative findings are:

  • Ease of Deployment and Use: Anafora is designed to be lightweight and easy to set up and use, even for users with limited technical expertise.
  • Flexibility for Complex Schemas: The tool's schema definition capabilities are significantly more advanced than many existing solutions, enabling the annotation of complex linguistic phenomena.
  • Improved Administration: Centralized storage and automated features streamline project management for administrators.
  • Support for Adjudication: The tool includes a robust adjudication process to resolve disagreements between annotators and establish gold-standard data.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Limited Support for Tree-Based Annotation: Anafora does not currently support tree-based annotation structures, which are common in tasks like constituent parsing or semantic role labeling.
  • Challenges with Complex Relation Linking: While it supports relations, linking relations to other relations is not directly supported.
  • Displaying Additional Text Information: Integrating external text information (e.g., Frame files, WordNet ontologies) alongside annotations within the same interface is challenging due to the tool's whole-text annotation design.

Future work includes:

  • Enhanced Annotation Efficiency and UX Design: Continued development to improve the user experience and speed up the annotation process.
  • Administrator Interface: Implementing a dedicated interface for managing users, tasks, and schemas.
  • Automated Pre-annotation: Integrating pre-annotation capabilities to speed up the initial annotation phase.
  • Comparison of Annotator Work: Developing features to compare annotator performance against gold data and their own past work.
  • Active Learning Integration: Exploring active learning strategies to improve annotation efficiency.
  • Comparison to Gold Standard Data: Allowing annotators to compare their work directly with gold standard data.

Summary updated

BibTeX
@inproceedings{chen2013anafora,
  author = {Wei-Te Chen and Will Styler},
  title = {{Anafora: A Web-based General Purpose Annotation Tool}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure