TS-ANNO: An Annotation Tool to Build, Annotate and Evaluate Text Simplification Corpora

Regina Stodden, Laura KallmeyerACL 2022

TL;DR

This paper introduces TS-ANNO, an open-source web application designed to streamline the creation and evaluation of text simplification (TS) corpora. It addresses the scarcity of high-quality, annotated TS datasets by providing a unified platform for manual alignment, annotation of simplification transformations, and rating of simplified text quality, ultimately aiming to improve TS systems.

What Problem Does This Paper Address?

Text simplification (TS) aims to make complex texts more accessible to a wider audience, including individuals with low literacy skills or cognitive disabilities. Developing effective TS systems relies heavily on parallel corpora – pairs of original and simplified sentences. However, existing TS corpora are often small, lack detailed annotations of simplification strategies, or suffer from unreliable automatic sentence alignment. This gap hinders the training and evaluation of robust TS systems. The paper tackles this by providing a comprehensive tool to facilitate the creation of high-quality, manually annotated TS corpora.

Methods

TS-ANNO is a web-based application built using Python (Django) and PostgreSQL. Its core functionalities are:

  • Data Ingestion: Supports both local file uploads (plain text, pre-aligned text) and web crawling to gather parallel documents from websites. Metadata like copyright and domain information can be added during upload.
  • Pre-processing: Utilizes SpaCy's NLP pipeline for language-specific pre-processing, including sentence splitting and tokenization.
  • Annotation:
    • Sentence Alignment: Allows for manual alignment of sentences, supporting n:m relationships (one-to-many, many-to-one, many-to-many). It also automatically aligns identical sentences and handles unaligned sentences after manual alignment.
  • Transformation Annotation: Enables annotators to mark specific tokens or phrases within an alignment pair and assign fine-grained simplification transformation labels (e.g., deletion, insertion, lexical substitution, sentence splitting) at word, phrase, sentence, or paragraph levels.
  • Rating: Supports both relative and absolute rating of aligned sentence pairs on Likert scales for aspects like grammaticality, simplicity, coherence, and meaning preservation. The rating aspects and scales can be customized.
  • Manual Simplification: Offers a sentence-wise manual simplification feature, allowing users to create simplified versions of complex sentences, useful for generating alternative simplifications or for domains not covered by existing corpora.
  • Evaluation: Calculates inter-annotator agreement (IAA) for alignments and ratings using Cohen's Kappa or Fleiss' Kappa, providing a measure of annotation consistency.
  • Data Export: Allows export of aligned sentence pairs, annotated information (ratings, transformations), and metadata in various formats (e.g., CSV, parallel files) to support downstream tasks.

Datasets

The paper mentions the use of existing datasets for demonstration and testing, including:

  • ASSET corpus: Used to demonstrate pre-aligned data upload and as a basis for further analysis.
  • Alumniportal Deutschland: Openly licensed, parallel German documents (CEFR B1-B2 complex, A1-A2 simple) used for testing the web crawler and for demonstration of annotation functionalities.
  • Inclusion Europe: Provides parallel complex-simple documents in four languages, also used for testing the web crawler.
  • wikiHow: Mentioned as a source for manual simplification examples, demonstrating the tool's applicability to various domains.

TS-ANNO itself is a tool for creating new datasets, rather than a dataset itself.

Key Insights and Ideas

  • Unified Platform: The most significant insight is the development of a single, integrated tool that covers the entire pipeline of TS corpus creation and evaluation, from data acquisition to annotation and quality assessment. This addresses the fragmentation of existing tools and processes.
  • Flexibility in Alignment: Supporting n:m sentence alignments is crucial, as text simplification often involves sentence splitting or merging, which 1:1 alignment methods cannot capture.
  • Fine-grained Transformation Annotation: The ability to annotate specific rewriting transformations at various levels (word, phrase, sentence, paragraph) provides rich information for understanding and improving TS systems, going beyond simple alignment or rating.
  • Subjectivity of Simplicity: The paper acknowledges that simplicity is subjective and can vary between annotators. TS-ANNO's rating features and the emphasis on metadata export (including annotator demographics) aim to address this by allowing for analysis of how annotator characteristics influence ratings.
  • Importance of Manual Annotation: The authors emphasize that while automatic alignment tools exist, their quality is often questionable, especially for "strong paraphrases." Manual alignment or post-editing is highlighted as essential for high-quality corpora.

Key Results

The paper focuses on the functionalities and design of the TS-ANNO tool rather than presenting quantitative results from large-scale corpus creation or system evaluation. The "results" are demonstrated through:

  • Successful Testing: TS-ANNO was tested with German parallel web texts from Alumniportal Deutschland, involving manual alignment and annotation of transformations and ratings.
  • Demonstration Version: A demo version of TS-ANNO is available, showcasing various alignment types, ratings, and transformations from the Alumniportal corpus.
  • Open-Source Availability: The tool is open-source, encouraging community contributions and wider adoption.

Limitations and Future Work

Limitations:

  • Language Support: Currently tested for English, Spanish, French, Farsi, and German. While SpaCy supports many languages, quality may vary, and for unsupported languages, pre-processing (sentence splitting, tokenization) is required.
  • No Active Learning: The tool does not currently support active learning, focusing on high-quality manual annotation.
  • Rating vs. Comprehension: The paper notes that rating alone might not be sufficient for evaluating TS texts; comprehension tests, reading time, and eye-fixation studies are also important but not integrated into TS-ANNO.
  • Subjectivity of Ratings: While addressed through customizable scales and metadata, the inherent subjectivity of simplicity ratings remains a challenge.

Future Work:

  • Highlighting Complex Phrases: Extend the simplification option to highlight complex phrases for easier identification.
  • Active Learning/Neural Alignment: Integrate active learning or neural alignment systems to suggest potential sentence alignments.
  • Expanded Web Crawler: Add more websites to the crawler to facilitate corpus creation in other languages and domains.
  • Disagreement Metric: Implement a disagreement metric to analyze how annotator characteristics relate to annotation choices, similar to polarization indices.
  • Integration of Comprehension Tests: Explore ways to integrate comprehension tests or other evaluation methods beyond simple rating.

Summary updated

BibTeX
@inproceedings{stodden2022ts,
  author = {Regina Stodden and Laura Kallmeyer},
  title = {{TS-ANNO: An Annotation Tool to Build, Annotate and Evaluate Text Simplification Corpora}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure