ET: A Workstation for Querying, Editing and Evaluating Annotated Corpora

Elvis de Souza, Cláudia FreitasEMNLP 2021

TL;DR

This paper introduces ET, a comprehensive workstation designed to streamline linguistic research and Natural Language Processing (NLP) tasks involving annotated corpora in the CoNLL-U format. By integrating powerful querying, editing, and quality evaluation tools, ET aims to make complex corpus manipulation accessible to researchers, even those without extensive programming experience.

What Problem Does This Paper Address?

Annotated corpora are fundamental for training and evaluating NLP systems. However, manipulating these corpora, especially those in formats like CoNLL-U (used by Universal Dependencies), can be a labor-intensive and technically challenging task for human researchers. Existing tools often focus on either annotation or analysis, but rarely offer a unified environment for querying, editing, and assessing the quality of these rich linguistic datasets. This gap makes it difficult for researchers to efficiently explore, refine, and validate their annotated corpora, hindering progress in linguistic research and NLP development.

Methods

ET is built around two core integrated web-browser-based environments:

  • Interrogatório: This environment focuses on querying and editing annotated corpora. It leverages Python CGI for the backend and JQuery for the frontend. Key features include:
  • Flexible Querying: Supports simple keyword searches, complex query expressions built via a GUI builder, and even custom Python scripts for advanced users. Queries can be filtered and sorted based on various annotation attributes (POS tags, dependency labels, etc.).
  • Editing Capabilities: Allows users to directly edit sentences, including modifying segmentation, adding/removing tokens, and correcting annotations. It also supports batch correction via user-edited Python scripts.
  • Corpus Management: Users can upload and manage CoNLL-U or raw text files, which are then processed by UDPipe for tokenization, tagging, and parsing if necessary.
  • Julgamento: This environment is dedicated to evaluating the quality of annotated corpora. It uses Python Flask for the backend. Its evaluation methods include:
  • N-gram Analysis: Identifies inconsistencies in dependency relations for similar n-grams across sentences.
  • Validation Scripts: Utilizes official Universal Dependencies scripts and custom scripts for Portuguese grammar rules to check for format correctness and annotation consistency.
  • Corpus Comparison: Allows for the comparison of two different annotations of the same sentences (e.g., system vs. gold-standard, or two annotators) using metrics from CoNLL evaluation scripts, sentence accuracy, and confusion matrices.

Datasets

The paper does not explicitly list specific datasets used for training ET itself. However, it mentions that ET is designed to work with corpora annotated in the CoNLL-U format, which is the standard for the Universal Dependencies (UD) project. The authors mention using the "bosque-ud-2.6" corpus for demonstration purposes.

Key Insights and Ideas

  • Unified Workflow: The most significant contribution is the integration of querying, editing, and evaluation into a single, accessible workstation. This streamlines the entire process of working with annotated corpora.
  • Accessibility for Non-Developers: ET's design prioritizes ease of use, with a GUI query builder and clear interfaces, making it usable by linguists and researchers without deep programming expertise.
  • Flexibility and Extensibility: While offering user-friendly interfaces, ET also provides advanced options like custom Python scripting for querying and batch correction, catering to a wide range of user needs.
  • Focus on Quality: The Julgamento module's emphasis on various evaluation methods, including n-gram analysis and comparative metrics, highlights the importance of robust quality assessment in corpus linguistics.
  • Dynamic Querying: ET's architecture, which avoids pre-indexing, allows for dynamic corpus changes and real-time querying, though this can impact query speed for very large corpora.

Key Results

The paper focuses on the functionality and design of ET rather than presenting quantitative results from extensive experiments. The "results" are demonstrated through the capabilities of the workstation itself:

  • Successful Integration: ET has been intensively used by the authors' research group for over two years, serving as the chosen framework for linguistic and NLP studies.
  • Demonstrated Functionality: The paper provides detailed explanations and screenshots of how users can query, edit, and evaluate corpora using Interrogatório and Julgamento.
  • Accessibility: Live demonstrations are available online, allowing users to explore ET without prior installation.

Limitations and Future Work

  • Installation: While the system is open-source, installing the server-side requires intermediate technology skills. The paper notes that testing was primarily done on Ubuntu distributions.
  • Query Speed: The decision to avoid pre-indexing, while enabling dynamism, means that queries on very large corpora might be slower compared to indexed systems.
  • Future Work: The authors suggest that ET could be expanded to further integrate functionalities of both corpus analysis and annotation tools, increasing its range of capabilities without requiring users to leave the workstation.

Summary updated

BibTeX
@inproceedings{souza2021et,
  author = {Elvis de Souza and Cláudia Freitas},
  title = {{ET: A Workstation for Querying, Editing and Evaluating Annotated Corpora}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
}
Figure