Appropriately Handled Prosodic Breaks Help PCFG Parsing

Zhongqiang Huang, Mary HarperNAACL 2010

TL;DR

This paper shows that prosodic cues, specifically "break indexes" from TOBI labeling, can significantly improve speech parsing accuracy. The key insight is that while directly integrating these breaks can be detrimental due to misalignments, a rescoring approach that restricts the search space to the best syntactic parses makes prosodic information highly effective.

What Problem Does This Paper Address?

Parsing spoken language is challenging because spontaneous speech is messy, containing disfluencies, hesitations, and variations not found in carefully crafted text. While syntax provides a strong backbone for understanding sentence structure, it's not always enough to resolve ambiguities. Prosody – the rhythm, stress, and intonation of speech – offers additional cues that can help disambiguate meaning, identify sentence boundaries, and signal important information. However, effectively integrating these prosodic cues into parsing models has been difficult. Previous attempts have either failed to improve performance or even degraded it, often attributed to misalignments between automatically detected prosodic breaks and true syntactic phrase boundaries. This paper aims to find a way to leverage prosodic information without being misled by these errors.

Methods

The authors explore three methods for integrating prosodic break indexes (from the TOBI system, which labels intonational breaks, hesitations, and other pauses) into PCFG (Probabilistic Context-Free Grammar) parsing models:

  • BRKINSERT: Treats prosodic breaks as additional tokens inserted into the parse tree at high nonterminal nodes. This is similar to adding punctuation.
  • BRKPOS: Models breaks as a sequence of observations parallel to the words, conditionally dependent on preterminal symbols. This approach is inspired by previous work that uses latent variables.
  • BRKPHRASE: Also models breaks as parallel observations but associates them with the highest nonterminal covering the preceding word. This method aims to avoid breaking n-gram dependencies and directly models the relationship between breaks and phrases.

A crucial part of their work involves using PCFG-LA (PCFG with Latent Annotations) models, which split nonterminals into latent tags to learn more complex dependencies.

The core contribution lies in their rescoring approaches:

  • ORACLE-RESCORE: Trains prosodically enriched grammars on "cleaned-up" data where misclassified breaks are corrected to 1 (a minor break). Then, it uses these grammars to rescore the n-best parses from a baseline syntactic model, biasing against potentially misclassified breaks in the test data.
  • DIRECT-RESCORE: Trains prosodically enriched models using unmodified, automatic breaks. These models are then used to rescore the n-best lists from a baseline syntactic model, effectively filtering out parses that are unlikely given the prosodic information.

Datasets

  • Fisher Corpus: Used for both training and evaluation. The evaluation set was split into development (first 1,020 trees) and test (remaining 3,917 trees). The main training data came from the Fisher dev1 and dev2 sets (16,519 trees).
  • Switchboard Corpus: Used as an additional training source to evaluate the effect of training data size.

Key Insights and Ideas

  • "Bad" breaks are harmful: Prosodic breaks that don't align with true phrase boundaries can actively hurt parsing performance, even more than the benefit gained from correctly identified breaks. This is especially true when breaks are directly inserted into the parse structure (BRKINSERT).
  • Latent variables help, but don't solve the break problem alone: While PCFG-LA models significantly improve parsing accuracy by learning complex dependencies, they don't fully overcome the negative impact of misclassified prosodic breaks.
  • Restricting the search space is key: The most significant improvements come from rescoring the n-best lists generated by a purely syntactic model. This strategy allows the prosodic information to guide the selection of the best parse without being solely responsible for generating it, thus mitigating the impact of erroneous breaks.
  • BRKPHRASE is promising: Among the direct integration methods, BRKPHRASE showed the most benefit, especially when combined with rescoring, as it models break-phrase relationships without disrupting grammar dependencies.

Key Results

  • Basic PCFGs: BRKPHRASE improved accuracy from 64.9% to 67.2% over the baseline. BRKINSERT also showed improvement (64.9% to 66.2%), while BRKPOS performed slightly worse.
  • PCFG-LA models (DIRECT parsing): Prosodic enrichment offered only minor, often insignificant, improvements over the baseline PCFG-LA model (e.g., BRKPHRASE only 0.1% better, BRKINSERT 0.7% worse).
  • PCFG-LA models with Rescoring: Both ORACLE-RESCORE and DIRECT-RESCORE methods led to significant improvements. The previously underperforming BRKINSERT model became 0.3-0.5% better than the baseline, and the best models (BRKPOS and BRKPHRASE) were 0.6-0.9% better.
  • Augmented Training Data: Training on a combination of Fisher and Switchboard data boosted all models, with prosodically enriched models showing significant gains, especially with rescoring.

Limitations and Future Work

  • Misaligned breaks: The primary limitation acknowledged is the issue of automatically detected breaks not aligning with true phrase boundaries, which can degrade performance.
  • Oracle experiment: While the oracle experiment (where bad breaks are corrected) showed significant gains, the BRKINSERT method was still less effective, suggesting that broken n-gram dependencies also play a role.
  • Future Work: The authors suggest that integrating TOBI classification with syntactic parsing could improve both tasks. They also note that while their rescoring approach is effective, further work could explore how to better integrate prosodic information directly into the parsing process without relying on external n-best lists. The paper also hints that fixing errors in break classification itself could yield even greater gains.

Summary updated

BibTeX
@inproceedings{huang2010appropriately,
  author = {Zhongqiang Huang and Mary Harper},
  title = {{Appropriately Handled Prosodic Breaks Help PCFG Parsing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}

Loading related content...

Figure