TL;DR
This paper introduces the PDTB Annotator, a Java-based tool designed to streamline the annotation and adjudication of discourse relations according to the Penn Discourse TreeBank (PDTB) framework. It addresses the need for an accessible, platform-independent tool that supports multiple languages and customizable annotation schemes, making PDTB-style annotation more feasible for new projects.
What Problem Does This Paper Address?
Discourse relations, which describe how sentences and clauses connect logically (e.g., cause-effect, contrast), are crucial for understanding text meaning and structure. The Penn Discourse TreeBank (PDTB) provides a widely adopted framework for annotating these relations. However, creating and managing these annotations can be labor-intensive. Existing tools were either too specialized, tied to specific theoretical frameworks (like Rhetorical Structure Theory), or required significant customization. This left a gap for a user-friendly, flexible, and openly available tool specifically tailored for the PDTB framework, especially for researchers starting new annotation projects or working with diverse languages.
Methods
The PDTB Annotator is a Java application designed for ease of use and cross-platform compatibility. Its core functionality revolves around a graphical user interface with three main panels:
- Relation List: Displays all annotated tokens for a given text, allowing users to add, undo, or reject tokens.
- Relation Editor: Facilitates the annotation of discourse relation features, including relation type, arguments (Arg1 and Arg2), senses, and comments.
- Raw Text: Presents the text being annotated, enabling users to select text spans (even discontinuous ones) for relation annotation.
The tool supports explicit and implicit discourse connectives and allows for custom sense hierarchies and implicit connective lists via a configuration file (Options.cfg). A key feature is its adjudication capability, where it can compare annotations from two annotators, highlight agreements and disagreements, and facilitate the creation of "gold standard" annotations. Annotations are stored in a simple, pipe-delimited text format for easy processing.
Datasets
The paper does not explicitly mention specific datasets used for training the annotator itself. However, it discusses the PDTB framework and mentions its use in various comparable corpora, including:
- Hindi Discourse Relation Bank
- Leeds Arabic Discourse TreeBank
- Biomedical Discourse Relation Bank
- Chinese Discourse TreeBank
- Turkish Discourse Bank
- Prague Dependency Treebank 3.0 (discourse layer)
- TED-Multilingual Discourse Bank (TED-MDB)
The PDTB Annotator is designed to work with raw text files (UTF-8 encoded) and produce annotation files in a specific format, which can then be used to build or extend these types of corpora.
Key Insights and Ideas
- Democratizing Discourse Annotation: The PDTB Annotator significantly lowers the barrier to entry for researchers wanting to annotate discourse relations using the PDTB framework, especially for those with limited resources or time.
- Flexibility is Key: The ability to customize sense hierarchies and implicit connectives is crucial for adapting the PDTB framework to different domains, languages, and research questions.
- Adjudication as a First-Class Citizen: Integrating adjudication directly into the annotation tool streamlines the process of creating high-quality, reliable datasets by facilitating conflict resolution between annotators.
- Simplicity in Output: The pipe-delimited text format for annotations is a pragmatic choice, making it easy to parse, process, and integrate with other NLP tools or spreadsheets, avoiding the complexities of XML for initial stages.
- Cross-Platform and Multi-Lingual Support: Being Java-based and supporting UTF-8 encoded text files makes the tool accessible to a broad range of users and applicable to a wide variety of languages.
Key Results
The paper is primarily a demonstration of a tool, not an empirical study with quantitative results. The "key results" are the features and benefits of the PDTB Annotator itself:
- Platform Independence: Runs on Mac, Windows, and Linux.
- Minimal Setup: Requires only text files and a configuration file.
- Multi-Lingual Support: Works with various writing systems due to UTF-8 encoding.
- Customizable Senses: Users can define their own sense hierarchy.
- Integrated Adjudication: Facilitates comparison and resolution of annotations between annotators.
- Simple Output Format: Pipe-delimited text files are easy to process.
Limitations and Future Work
- Adjudication for Two Annotators: The current adjudication feature is designed for comparing annotations from at most two annotators. The authors suggest that extending this to handle more annotators would be a valuable future direction.
- Task Complexity Study: The authors note that new features for recording adjudication reasons and disagreement types can be directly used to study task complexity in greater depth, implying this is an area for future research enabled by the tool.
- Tool Availability: The tool is available at
http://www.seas.upenn.edu/~pdtb/annotator.html.
Summary updated