UMR-Writer 2.0: Incorporating a New Keyboard Interface and Workflow into UMR-Writer

Sijia Ge, Jin Zhao, Kristin Wright-bettner, Skatje Myers, Nianwen Xue, Martha PalmerACL 2023

TL;DR

UMR-Writer 2.0 introduces a new keyboard-centric interface and an improved project management workflow for annotating Uniform Meaning Representation (UMR) graphs. This aims to significantly speed up annotation for experienced users and address usability issues in the previous click-based system, while also providing better tools for managing large-scale annotation projects.

What Problem Does This Paper Address?

Annotating complex semantic representations like Uniform Meaning Representation (UMR) can be time-consuming and prone to errors. The original UMR-Writer tool relied on a click-based interface, which, while intuitive for beginners, presented several challenges for experienced annotators. These included a need for excessive mouse movements, confusing drop-down menus, and difficulties in annotating non-sequential text spans or concepts that don't directly map to contiguous words. Furthermore, managing annotation data for multiple corpora and projects was becoming cumbersome as the scale of annotation efforts grew.

Methods

The core innovation is the introduction of a keyboard-based interface for UMR-Writer 2.0. This interface draws inspiration from the established AMR editor, allowing annotators to input commands directly using a keyboard. Key aspects include:

  • Index-based Command Input: Tokens in the raw text are assigned numerical superscripts (1-based indexing). Annotators use these indices, prefixed with 'x', to refer to tokens in commands (e.g., x1 for the first token).
  • Sense and Relation Specification: Commands allow for specifying concept senses (e.g., x3-01 for the first sense of the concept associated with the third token) and relations (e.g., :ARG1).
  • Handling Non-Sequential Spans: The interface supports annotating concepts composed of non-sequential tokens by concatenating their indices (e.g., x5_x7).
  • Direct Frame Information Display: The keyboard interface frees up screen real estate previously occupied by drop-down menus, allowing for direct display of frame information, reducing cognitive load.
  • Variable Naming Convention: Variables used to represent nodes in UMR graphs are now index-based (e.g., s2x3) to ensure consistency across languages and avoid ambiguity with initial-based variables. Abstract concepts are marked with ac (e.g., s2ac15).

Additionally, the paper introduces a project management workflow to handle annotation data for multiple corpora more efficiently. This includes:

  • Project Folders: Organizing data into distinct project folders.
  • Permission Hierarchy: A system of roles (admin, edit, annotate, view) to control access and actions within projects.
  • Quality Control (QC) and Double Annotated (DA) Sub-folders: Dedicated spaces for storing final annotations and multiple annotations of the same file for comparison.
  • Efficient Workflow: Defining clear steps for creating projects, adding members, assigning permissions, checking out files, and submitting annotations.

Datasets

Not discussed in the paper. The paper focuses on the tool and its workflow, not on training or evaluating models on specific datasets.

Key Insights and Ideas

  • Keyboard-centric annotation is faster and more efficient for experienced users: By mimicking the command-line style of the AMR editor, UMR-Writer 2.0 caters to users already familiar with such interfaces, reducing the learning curve and increasing annotation speed.
  • Directly displaying frame information significantly improves usability: Eliminating the need to hover over elements to see details reduces mouse movement and streamlines the annotation process.
  • Handling non-sequential tokens is crucial for complex linguistic phenomena: The ability to easily annotate concepts spread across non-contiguous words is a significant improvement for capturing nuanced linguistic structures.
  • Robust project management is essential for large-scale annotation: The introduced workflow and permission system address the practical challenges of coordinating multiple annotators and corpora, ensuring data integrity and quality.
  • Cross-lingual consistency is a design priority: The shift to index-based variables for UMR graphs aims to create a more universal and less ambiguous annotation system across different languages.

Key Results

The paper highlights that the new keyboard interface reduces the annotation process for adding a single node from five steps in the click-based interface to a single command. This directly translates to increased efficiency and potentially higher accuracy due to fewer opportunities for errors. The project management workflow has been successfully applied to several projects, including the THYME corpus and the Arabic UMR corpus, demonstrating its practical utility.

Limitations and Future Work

  • Low-Resource Language Challenges: While the click-based interface supports low-resource languages, the keyboard interface currently requires annotators to manually count character indices in tokens for morphologically complex languages like Arapaho, which is less ideal. Future work aims to simplify this process.
  • Segmentation Errors: For languages with ambiguous word boundaries (e.g., Chinese), raw text segmentation can be problematic. Annotators currently use underscores to concatenate tokens, which is inconvenient. Future work plans to allow manual correction of segmentation errors by adding/deleting spaces.
  • Eventive Concept Detection: The tool currently lacks a feature to detect eventive concepts automatically. The authors plan to develop a system for this, which would also provide auto-complete reminders to assist annotators.
  • Refactoring and UI Improvements: Future plans include refactoring the codebase into a JavaScript framework like React.js, improving document-level annotation visualization, and implementing auto-completion for commands.
  • Wikidata Integration: Mapping UMR named entity hierarchies to Wikidata ontologies is another planned future direction.

Summary updated

BibTeX
@inproceedings{ge2023umr,
  author = {Sijia Ge and Jin Zhao and Kristin Wright-bettner and Skatje Myers and Nianwen Xue and Martha Palmer},
  title = {{UMR-Writer 2.0: Incorporating a New Keyboard Interface and Workflow into UMR-Writer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure