Coordination Annotation Extension in the Penn Tree Bank

Jessica Ficler, Yoav Goldberg

This paper introduces a significant enhancement to the Penn Treebank (PTB) by providing detailed, hierarchical annotations for coordination structures. By explicitly labeling functional roles like conjuncts and shared modifiers, the authors provide a cleaner, more consistent dataset that improves parsing accuracy for one of language's most frequent yet challenging syntactic constructions.

TL;DR

The authors present a manual annotation extension for the Penn Treebank that fixes inconsistent and "flat" coordination structures. By adding internal hierarchy and explicit function labels (like conjuncts vs. shared modifiers), they provide a high-quality resource that improves state-of-the-art parsing performance and highlights exactly where current models struggle with complex syntax.

What Problem Does This Paper Address?

Coordination—the linking of words or phrases using conjunctions like "and," "or," or "but"—is ubiquitous, appearing in nearly 40% of sentences in the Penn Treebank. Despite its frequency, it remains a major hurdle for natural language processing.

The core issue lies in the PTB's original annotation style. In many cases, especially within Noun Phrases (NPs), the structure is "flat." For example, a phrase like "chairman and chief executive officer" might be represented as a single level of siblings, making it impossible for a parser to programmatically determine where one title ends and the next begins. Furthermore, the PTB often fails to distinguish between the actual items being coordinated (conjuncts) and other elements like markers ("both," "either"), connectives ("and thus"), or shared modifiers ("Venus’s density and temperature"). This lack of structural depth has historically forced researchers to use smaller, niche datasets rather than the industry-standard PTB.

Methods

The authors combined automatic rules with rigorous manual annotation to enrich the PTB. Their approach focused on two main pillars:

  1. Explicit Function Marking: They introduced a new labeling scheme using a -XXX suffix. Phrases containing coordination are marked with CCP. Within these, nodes are assigned specific roles:
  2. COORD: The actual elements being joined.
  3. CC: The coordinator (e.g., "and").
  4. MARK: Markers like "both" or "neither."
    • CONN: Connectives or parentheticals.
  5. SHARED: Modifiers or arguments that apply to all conjuncts.
  6. Structural Refinement: They moved away from flat structures by adding brackets to delimit element spans. If a conjunct was previously split across multiple nodes due to an error, they consolidated it.
Comparison of a flat NP in the original PTB (top) vs. the new hierarchical structure with function labels (bottom).

The process involved automatically identifying CCP candidates and then having a dedicated annotator manually define the spans for roughly 3,500 complex cases.

Datasets

The work is built directly upon the Penn Treebank (PTB). Specifically, the authors used the version that includes Vadas and Curran’s (2007) base-NP bracketing as their starting point. The final output is released as a "diff" over the original PTB, allowing other researchers to easily integrate these improved annotations into their existing pipelines.

Key Insights and Ideas

  • Flatness is a Feature Gap: The authors argue that the "flat" annotation style isn't just a simplification; it's a data deficiency that prevents parsers from learning the nuances of English syntax.
  • The "Shared Element" Challenge: One of the most insightful findings is the difficulty of identifying shared modifiers. In a phrase like "the economy's ups and downs," "the economy's" modifies both "ups" and "downs." Standard parsers often fail to recognize this shared relationship, either attaching it only to the first conjunct or excluding it from the coordination structure entirely.
  • Consistency Over Novelty: The authors prioritized staying as close to the original PTB structures as possible, only making changes where internal structure was missing or clearly erroneous. This makes the dataset highly backwards-compatible.

Key Results

The researchers evaluated their new annotations using the state-of-the-art Berkeley parser.

  • Quantitative Gains: When trained and tested on the enhanced data (PTB+NPB+CCP), the parser's F1 score on section 22 improved from 88.21 to 91.01. While the authors note the test sets aren't perfectly identical due to the added constituents, the jump is significant.
  • Labeling Accuracy: For coordination phrases where the parser correctly predicted the structure, it assigned the correct function labels (conjunct, marker, etc.) 98.91% of the time.
  • Error Analysis: The confusion matrix (Table 4 in the paper) reveals that the hardest task for the parser is identifying SHARED elements; only about 75% of gold shared elements were correctly identified, suggesting a clear target for future algorithmic improvements.
A complex coordination structure showing the interplay between COORD, CONN, and SHARED labels.

Limitations and Future Work

The authors acknowledge that their work focuses primarily on elements inside the coordination phrase. Shared arguments that appear outside the phrase (like a subject governing a coordinated verb phrase) were not the focus of this specific extension.

Additionally, while inter-annotator agreement was high (98.1% after revision), some semantic ambiguities remain difficult to resolve definitively. The authors suggest that the list of ambiguous modifier attachments generated during this process could serve as a valuable resource for developing new, more nuanced error metrics for syntactic parsing.

Summary updated

Abstract

Coordination is an important and common syntactic construction which is not handled well by state of the art parsers. Coordinations in the Penn Treebank are missing internal structure in many cases, do not include explicit marking of the conjuncts and contain various errors and inconsistencies. In this work, we initiated manual annotation process for solving these issues. We identify the different elements in a coordination phrase and label each element with its function. We add phrase boundaries when these are missing, unify inconsistencies, and fix errors. The outcome is an extension of the PTB that includes consistent and detailed structures for coordinations. We make the coordination annotation publicly available, in hope that they will facilitate further research into coordination disambiguation.
BibTeX
@inproceedings{ficler2016coordination,
  author = {Jessica Ficler and Yoav Goldberg},
  title = {{Coordination Annotation Extension in the Penn Tree Bank}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/p16-1079},
}
Figure