Annotation Tool for Discourse in PDT

Jiří Mírovský, Lucie Mladová, Zdeněk ŽabokrtskýCOLING 2010

TL;DR

This paper introduces a specialized annotation tool designed to enrich the Prague Dependency Treebank (PDT) with semantic inter-sentential discourse relations. The tool offers a user-friendly interface with features like simultaneous display of parallel annotations and flexible argument specification, aiming to improve annotation efficiency and inter-annotator agreement for this complex linguistic task.

What Problem Does This Paper Address?

Linguistic corpora, like the Prague Dependency Treebank (PDT), are increasingly being annotated with detailed syntactic and morphological information. However, capturing the deeper semantic connections between sentences – discourse relations – presents a significant challenge. Manually annotating these relations is time-consuming and prone to inconsistencies. The authors aim to address this by developing a sophisticated annotation tool that streamlines the process of identifying and marking discourse relations on the tectogrammatical (deep syntactic) layer of the PDT, while also facilitating comparison with other annotation schemes like the Penn Treebank.

Methods

The core of the paper describes an annotation tool built as an extension for TrEd, a customizable XML-based tree editor. The tool leverages the existing PML format of the PDT and enhances it to support discourse relation annotation. Key features include:

  • Flexible Argument Specification: Discourse relations have two arguments (initial and target). The tool allows these arguments to be defined not just as entire subtrees but also as specific parts of subtrees, subsequent or preceding trees, or even arbitrary groups of nodes. This flexibility is crucial for accurately capturing the extent of discourse relations.
  • Simultaneous Parallel Annotations: To study and improve inter-annotator agreement, the tool can display annotations from multiple annotators side-by-side. This allows for direct comparison of how different annotators interpret and mark the same discourse phenomena.
  • Connective Representation: Connectives (like "however," "therefore") are represented by linking to specific nodes in the tectogrammatical or analytical layers, or even allowing identifiers from the analytical layer if a direct tectogrammatical node isn't suitable.
  • Collapsed Trees: For better visualization and to manage complexity, subtrees representing individual clauses can be collapsed into single nodes. This simplifies the display of large trees while still allowing for detailed annotation.
  • Interactive Annotation: Annotators can choose to mark relations directly on the tree structures or within a linear text view, where relevant tokens are highlighted.

Datasets

The primary dataset discussed is the Prague Dependency Treebank 2.0 (PDT 2.0), a manually annotated corpus of Czech. The paper focuses on enriching this existing dataset with semantic inter-sentential discourse relations on its tectogrammatical layer. The work also aims to compare the PDT annotation scheme with that of the Penn Treebank, specifically its discourse relation annotations.

Key Insights and Ideas

  • The Importance of Flexible Argument Definition: Capturing discourse relations accurately requires more than just pointing to a single node. The ability to define arguments as specific subtrees, groups of nodes, or even across sentence boundaries is critical for precise annotation.
  • Parallel Annotation as a Tool for Quality Control: Displaying multiple annotators' work simultaneously is a powerful method for identifying discrepancies, understanding sources of disagreement, and ultimately improving the consistency and reliability of the annotations.
  • Integration with Existing Treebank Structures: Building upon the tectogrammatical layer of PDT simplifies annotation by leveraging pre-existing syntactic information, resolving some ambiguities before discourse annotation even begins.

Key Results

The paper primarily describes the features and functionality of the annotation tool rather than presenting quantitative evaluation results of the annotations themselves. The key "results" are the successful implementation of a tool that:

  • Enables the annotation of semantic inter-sentential discourse relations on the PDT's tectogrammatical layer.
  • Offers a user-friendly interface with features designed to increase efficiency and accuracy.
  • Provides a mode for studying parallel annotations, aiding in the analysis of inter-annotator agreement.

Limitations and Future Work

  • Evaluation Difficulty: The authors acknowledge that evaluating such a specialized tool is difficult because it doesn't produce direct results beyond the annotated data itself. Its effectiveness is tied to the specific needs of the PDT and its annotation framework.
  • Future Work: The paper mentions that the annotation framework is based on knowledge gained from studying other systems (like the Penn Discourse Treebank) and has been adjusted for Czech and PDT's specific needs. Future work would likely involve further refinement of the tool and the annotation guidelines based on ongoing annotation efforts and analysis of inter-annotator agreement.

Summary updated

BibTeX
@inproceedings{mirovsky2010annotation,
  author = {Jiří Mírovský and Lucie Mladová and Zdeněk Žabokrtský},
  title = {{Annotation Tool for Discourse in PDT}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}
Figure