Comma Restoration Using Constituency Information

Stuart M. Shieber, Xiaopeng TaoNAACL 2003

TL;DR

This paper explores how syntactic information, specifically constituency structure, can significantly improve the automatic restoration of commas in unpunctuated text. By integrating this linguistic knowledge into a standard HMM-based approach, the authors achieve a substantial reduction in sentence error rates, demonstrating the practical value of parsing for end-user applications.

What Problem Does This Paper Address?

Modern speech recognition systems have become quite adept at transcribing spoken words, but they often require users to explicitly dictate punctuation. This can be cumbersome and unnatural. To create a more seamless user experience, systems need to automatically reconstruct punctuation, like commas, from the word sequence. While sentence boundary punctuation (periods, question marks) is relatively well-studied, restoring intra-sentence punctuation, particularly commas, is a less explored area. The challenge lies in modeling the subtle grammatical roles commas play, such as marking constituents or separating conjoined elements, which are difficult to capture with simple n-gram language models alone.

Methods

The authors build upon a baseline HMM (Hidden Markov Model) approach, specifically a three-state HMM, that models comma restoration using trigram probabilities of word sequences. This baseline, a replication of the CYBERPUNC system, achieves a sentence accuracy of 47%.

The core innovation is the integration of syntactic constituency information into this HMM framework. They explore several ways to leverage this:

  • Constituent Start/End Counts (sc/ec): They define metrics for how many constituents begin (sc) or end (ec) at a given position in a sentence's parse tree. They hypothesize that comma placement correlates with these counts.
  • Thresholded Constituency Information: They introduce binary features indicating whether the ec count exceeds a certain threshold (e.g., ec >= 3). This simplified syntactic information is then incorporated as an additional state in the HMM.
  • Full Constituency Information: They also experiment with using the raw ec values directly, employing a smoothing technique to handle data sparsity.
  • Part-of-Speech (POS) Information: They further augment the model by incorporating POS tags derived from the parse trees.

Crucially, they investigate the impact of using parses from different sources: the "oracular" Treebank parses (considered ground truth) versus parses generated by an automated statistical parser (Collins's parser). This allows them to assess how well their methods generalize when using imperfect, automatically derived syntactic information.

Datasets

The primary dataset used is the Wall Street Journal (WSJ) portion of the Penn Treebank.

  • Training: Sections 02-22 of the WSJ data were used for training the language models and constituency-based features.
  • Testing: Section 23 of the WSJ data was used for evaluating the models.

For consistency with prior work, they also used a slightly varied subset of the WSJ data (sections 02-23) that was employed in the original CYBERPUNC experiments.

Key Insights and Ideas

  • Syntactic Structure Matters for Commas: The paper strongly supports the intuition that comma placement is deeply tied to the grammatical structure of a sentence (constituency). Commas tend to appear at the boundaries of syntactic units, not in their middle.
  • Even Sparse Syntactic Cues Help Dramatically: Surprisingly, even very simplified syntactic information (like a single binary threshold on constituent end counts) can lead to significant improvements over a purely text-based model. This suggests that the type of information is more critical than its granularity for initial gains.
  • Automated Parsing is Sufficient for Improvement: While using perfect, manually annotated parses yields the best results, using parses from an automated statistical parser (Collins's parser) still provides substantial gains over the baseline. This is a critical finding, as it demonstrates the practical applicability of their approach to real-world, automatically processed text.
  • Prosody and Syntax are Likely Orthogonal: The authors suggest that their syntactic features are likely orthogonal to prosodic features (like speech duration or pitch), meaning they can be combined to achieve even greater improvements in comma restoration.

Key Results

The paper presents a series of models, with performance measured by precision, recall, F-measure, token accuracy, and sentence accuracy.

  • Baseline (Trigram HMM): Achieved a sentence accuracy of 47.0%.
  • Model 3 (Thresholded ec with Treebank parse): Significantly improved sentence accuracy to 52.3% (a ~30% reduction in error rate compared to the baseline).
  • Model 8 (Full ec with Treebank parse): Further improved F-measure to 72.1% and sentence accuracy to 57.9% (a ~28% improvement over the baseline).
  • Model 10 (POS + ec with Treebank parse): Achieved the best performance with Treebank parses, reaching an F-measure of 74.8% and sentence accuracy of 57.9%.
  • Model 5 (Thresholded ec with Collins parse): Demonstrated that using an automated parser, sentence accuracy improved to 54.9% (twice the improvement of the baseline).
  • Model 7 (Full ec with Collins parse): Achieved 66.8% F-measure and 50.7% sentence accuracy when using Collins parses.

Overall, their best methods reduced sentence error rate by approximately 20%, with a potential for an additional 8% reduction if automated parsers reach Treebank quality.

Limitations and Future Work

  • Data Sparsity: Using more detailed syntactic information (like raw ec counts) leads to data sparsity issues, which they address with smoothing techniques.
  • Reliance on Parsed Input: The current methods rely on having access to syntactic parses. For practical application, they acknowledge the need to use statistically generated parses, which they investigate with Collins's parser.
  • Ideal Transcriptions: The experiments assume ideal reference transcriptions. The performance on automatically transcribed speech, which contains errors, remains to be determined. They cite related work showing a significant drop in performance for prosody-based methods on noisy speech.
  • Future Directions:
    • Investigate the performance on automatically transcribed speech.
  • Explore alternative modeling techniques like maximum entropy models or transformation-based learning.
  • Combine syntactic features with prosodic features, as they are likely orthogonal.
  • Consider using the statistical parsing model itself as a model for comma placement, though this might lose longer-distance lexical relationships.

Summary updated

BibTeX
@inproceedings{shieber2003comma,
  author = {Stuart M. Shieber and Xiaopeng Tao},
  title = {{Comma Restoration Using Constituency Information}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure