NEU Systems in SIGHAN Bakeoff 2012

Ji Ma, LongFei Bai, Zhuo Liu, Ao Zhang, Jingbo ZhuCIPS-SIGHAN 2012

Introduction

Statistical parsing relies heavily on large, well-annotated treebanks to train machine learning models. While the Penn Chinese Treebank (CTB) is a widely used standard for Simplified Chinese, the Sinica Treebank provides a vital resource for Traditional Chinese. However, researchers have found parsing the Sinica Treebank to be unusually difficult. This paper details Northeastern University's approach to parsing the Traditional Chinese Sinica Treebank for the SIGHAN 2012 Bakeoff. Using an unlexicalized PCFG model with latent annotations, the authors discover that the treebank's highly fine-grained annotation scheme creates severe data sparsity, limiting parsing performance. They also highlight how specific part-of-speech (POS) tagging ambiguities drastically bottleneck syntactic parsing accuracy.

Methodology

The authors opted against lexicalized (head-driven) parsing models, such as the Collins or Charniak parsers, because lexicalized models require manually crafted head-finding rules which do not exist for the Sinica Treebank. Furthermore, such rules are usually over-optimized for the Penn Treebank format and perform poorly when shifted to new annotation schemes.

Instead, they used an unlexicalized PCFG with Latent Annotations (PCFGLA), specifically employing the Berkeley Parser. This method automatically learns latent annotations by iteratively splitting and merging non-terminal symbols using an Expectation-Maximization (EM) algorithm. This allows the model to capture the behavioral preferences of grammar rules (e.g., distinguishing between transitive and intransitive verb phrases) without relying on hard-coded linguistic rules. To combat the data sparsity inherent to the dataset, the authors also experimented with simplifying the grammar by stripping suffixes from POS tags and non-terminal labels prior to training.

Data Sparsity and Model Optimization

The Sinica Treebank has 101 POS tags compared to CTB's 42. It uses highly specific suffixes (e.g., annotating a verb as VC[+NEG]). This fine-grained approach causes severe data sparsity, making it difficult for statistical models to estimate parameters reliably because specific tags appear too infrequently.

The Berkeley parser learns by iteratively splitting grammar rules to make them more specific. While it usually peaks around the 6th split-merge iteration on Penn Treebanks, it peaked at the 4th iteration on the Sinica Treebank. Further iterations degraded performance, confirming that the model was starving for data and overfitting to the training set.

Parsing performance across different split-merge iterations of the Berkeley Parser.

Structural Ambiguity: The Ncd vs. Ng Trap

A significant finding was the impact of upstream POS tagging errors on downstream syntactic parsing. When the parser was fed "gold" (perfect, human-annotated) POS tags, the parsing F1 score jumped by over 6 points.

A specific bottleneck identified was the confusion between Ncd and Ng tags. Both represent positional information (e.g., "outside", "in"), but they trigger entirely different syntactic tree structures. Confusing them, which the parser frequently does, ruins the resulting parse tree.

Comparison of syntactic structures triggered by Ncd and Ng tags.

Key Results

  • Baseline Performance: The Berkeley Parser achieved a 78.30 F1 score and 91.58% POS tagging accuracy on the development set.
  • Suffix Removal: By removing suffixes from POS and non-terminal labels (reducing unique POS tags from 101 to 55), the F1 score slightly improved to 78.66.
  • Aggressive Mapping: Mapping all non-terminals to just 7 basic phrase labels hurt performance (F1 dropped to 75.69), indicating that some granularity is necessary for learning structural rules.
  • Gold POS Impact: Using Gold POS tags caused the F1 score to skyrocket to 84.69, highlighting tagging accuracy as the primary bottleneck.

Conclusion

The authors acknowledge that simply stripping suffixes or aggressively mapping labels is a blunt instrument. Their findings strongly suggest that future efforts on the Sinica Treebank must focus on joint POS-tagging and parsing models to reduce cascading errors. Additionally, developing better smoothing techniques or utilizing pre-trained word embeddings would be necessary to handle the extreme data sparsity caused by the fine-grained annotation scheme.

Summary updated

BibTeX
@inproceedings{ma2012neu,
  author = {Ji Ma and LongFei Bai and Zhuo Liu and Ao Zhang and Jingbo Zhu},
  title = {{NEU Systems in SIGHAN Bakeoff 2012}},
  booktitle = {CIPS-SIGHAN},
  year = {2012},
}
Figure