Sentence-Internal Prosody Does not Help Parsing the Way Punctuation Does

Michelle Gregory, Mark Johnson, Eugene CharniakNAACL 2004

TL;DR

This paper investigates whether prosodic cues in speech, similar to punctuation in text, can improve the accuracy of statistical parsers for transcribed speech. The study found that while punctuation significantly boosts parsing accuracy, incorporating prosodic information actually decreased it, suggesting current methods of encoding and utilizing prosody are not as effective as punctuation for this task.

What Problem Does This Paper Address?

Syntactic parsing, the process of determining the grammatical structure of a sentence, is crucial for many natural language processing tasks. In written text, punctuation marks like commas and periods provide valuable cues for parsers, helping them identify sentence boundaries and phrase structures. Intuitively, prosodic features in spoken language—such as pauses, pitch changes (f0), and speech duration—should serve a similar role, offering analogous information to guide parsing. However, the effectiveness of these prosodic cues for sentence-internal parsing (i.e., parsing within a sentence, not just at boundaries) remains less clear. This paper aims to bridge this gap by directly comparing the impact of punctuation and prosodic features on the performance of a statistical parser for transcribed speech.

Methods

The researchers adapted a standard statistical parser to incorporate prosodic information. They encoded prosodic features (duration, pausing, and f0) into "pseudo-punctuation" symbols, mirroring how punctuation is represented in the Penn Treebank (as terminal and preterminal nodes). This involved:

  • Data Preprocessing: Using transcribed speech from the LDC Switchboard corpus.
  • Prosodic Feature Extraction: Utilizing forced alignments and word-level prosodic information (pause duration, normalized rhyme duration, pitch variations, and pitch slope) from existing datasets.
  • Binning Continuous Features: Since the parser requires categorical input, continuous prosodic variables were binned into discrete categories. Different binning strategies were explored, with the number of bins having little impact on results.
  • Pseudo-Punctuation Generation: Combining binned prosodic features to create "pseudo-punctuation" symbols. These symbols were inserted into the parse input, attached to the tree structure in a manner similar to actual punctuation.
  • Parser Training and Evaluation: The parser was trained and tested on various versions of the corpus: with standard punctuation, with prosodic pseudo-punctuation, with both, and with neither. The evaluation used the standard PARSEVAL metrics, ignoring prosodic and punctuation preterminals.

Datasets

The primary dataset used was the LDC Switchboard Corpus, a collection of telephone conversations between adult speakers of varying dialects. This corpus was split into training, testing, and development sets as described in Charniak and Johnson (2001).

Key Insights and Ideas

  • Punctuation is a Strong Signal: The study confirms previous findings that punctuation significantly improves parsing accuracy, increasing the F-score by over 1 percentage point.
  • Prosody's Limited Impact (in this form): Despite the intuitive similarity between prosody and punctuation, incorporating prosodic information in the way punctuation is encoded decreased parsing accuracy.
  • Encoding Matters: The researchers hypothesize that their method of encoding prosodic information might not be optimal, or that the parser was not sufficiently tuned to utilize it.
  • Data Sparseness is a Hurdle: Adding prosodic symbols, like punctuation, can increase data sparseness. While punctuation provides enough signal to overcome this, prosodic cues, as encoded here, do not.
  • Human vs. Automatic Annotation: A key difference highlighted is that punctuation is human-annotated and thus reflects a deeper comprehension of the utterance, whereas prosodic features were automatically extracted and might be noisier or less informative for the parser.

Key Results

  • Baseline (no punctuation/prosody): F-score of 86.9%.
  • With Punctuation: F-score increased to 88.2%.
  • With Prosodic Information (various combinations): F-scores decreased to between 84.8% and 86.8%.
  • The general trend observed was that performance decreased as the amount and complexity of the prosodic annotation increased.

Limitations and Future Work

Limitations:

  • Encoding Method: The researchers acknowledge that their method of encoding prosodic information might not be optimal.
  • Parser Tuning: The parser was not specifically tuned to leverage the prosodic information.
  • Continuous vs. Categorical: The parser requires categorical input, forcing continuous prosodic features into bins, which inherently involves a loss of information. Developing a parser capable of handling continuous prosodic variables directly is a significant undertaking.
  • Noise in Automatic Extraction: Prosodic features were automatically extracted, potentially introducing noise compared to human-annotated punctuation.

Future Work:

  • Different Prosodic Cues: Exploring alternative prosodic cues that might be more informative for parsing.
  • Novel Exploitation Methods: Investigating different ways to encode and exploit prosodic information within the parser, beyond the direct analogy to punctuation.
  • Speech Recognition Output: The authors suggest that prosodic information might be more useful when parsing speech that comes directly from speech recognition systems, where word identity itself is uncertain, making prosody a more critical signal.

Summary updated

BibTeX
@inproceedings{gregory2004sentence,
  author = {Michelle Gregory and Mark Johnson and Eugene Charniak},
  title = {{Sentence-Internal Prosody Does not Help Parsing the Way Punctuation Does}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}

Loading related content...

Figure