SenTag: A Web-Based Tool for Semantic Annotation of Textual Documents

Andrea Loreggia, Simone Mosco, Alberto ZerbinatiAAAI 2022 · DOI

Introduction

In Natural Language Processing (NLP), the creation of high-quality "gold standard" datasets is a critical yet labor-intensive process. Manual annotation is frequently plagued by human errors, such as tag misspellings, incorrect attribute usage, and structural inconsistencies. Furthermore, modern NLP tasks often require capturing complex logical relationships—such as support or attack between claims—which are difficult to manage in traditional text editors. SenTag is a lightweight, web-based tool designed to address these challenges by integrating real-time XML schema validation and a novel graphical interface for argument mapping.

System Architecture and Methodology

SenTag is developed using a modern web stack, featuring a Python 3.9 and Django 3.2 backend paired with a Vue 3 frontend. The system implements a role-based workflow to manage annotation projects effectively:

  • Admins: Oversee user management.
  • Editors: Upload XML schemas and documents, assign tasks, and monitor progress.
  • Annotators: Perform the primary task of semantic tagging.

A core innovation of SenTag is its schema-driven validation. By requiring an XML schema at the project's outset, the tool enforces structural integrity during the annotation process, alerting users to missing attributes or invalid hierarchies before the data is exported.

Semantic Annotation Interface

The annotation interface allows users to highlight text segments and apply semantic tags defined by the project schema. This process is designed to be intuitive, abstracting the underlying XML complexity into a user-friendly visual format.

The SenTag annotation interface showing highlighted text segments and associated semantic tags.

Visual Argument Mapping

Beyond simple entity tagging, SenTag facilitates the creation of complex argument graphs. When a text segment is tagged as an "argument" or "claim," it is automatically represented as a node in a dedicated graph interface. Users can then visually define relationships (edges) between these nodes.

The graphical interface in SenTag allows users to visualize and define relationships between annotated claims and motives.

This visual approach ensures that the resulting relational data is well-formed and accurately reflects the logical flow of the document. The system handles the management of unique identifiers and relational attributes in the background, allowing annotators to focus on the semantic content.

Quality Assurance and Validation

To ensure data reliability, SenTag supports multi-user annotation of the same corpus. The platform automatically calculates Krippendorff’s alpha to measure inter-annotator agreement, providing editors with a quantitative metric to assess dataset quality. This integrated approach transforms annotation from a solitary task into a managed, quality-controlled project.

Conclusion and Future Work

SenTag successfully streamlines the semantic annotation pipeline by preventing structural errors at the point of entry and providing powerful tools for relational mapping. While the current iteration is purely manual, future developments aim to incorporate Artificial Intelligence (AI) assistance. By leveraging ensemble machine learning methods, the tool will eventually suggest tags, transitioning to a "human-in-the-loop" model that significantly accelerates the creation of high-quality NLP datasets.

Summary updated

BibTeX
@inproceedings{loreggia2022sentag,
  author = {Andrea Loreggia and Simone Mosco and Alberto Zerbinati},
  title = {{SenTag: A Web-Based Tool for Semantic Annotation of Textual Documents}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21724},
}
Figure