Better Punctuation Prediction with Dynamic Conditional Random Fields

Wei Lu, Hwee Tou NgEMNLP 2010 · OpenReview

Abstract

This paper focuses on the task of inserting punctuation symbols into transcribed conversational speech texts, without relying on prosodic cues. We investigate limitations associated with previous methods, and propose a novel approach based on dynamic conditional random fields. Different from previous work, our proposed approach is designed to jointly perform both sentence boundary and sentence type prediction, and punctuation prediction on speech utterances. We performed evaluations on a transcribed conversational speech domain consisting of both English and Chinese texts. Empirical results show that our method outperforms an approach based on linear-chain conditional random fields and other previous approaches.

Introduction

Automatic Speech Recognition (ASR) systems typically produce a continuous stream of words without capitalization or punctuation. This "word salad" output is difficult for humans to read and poses significant challenges for downstream Natural Language Processing (NLP) tasks such as machine translation, parsing, or part-of-speech tagging.

Traditionally, punctuation prediction relied heavily on prosodic cues—audio features like pitch and pause duration. However, in many NLP pipelines, only text is available. Existing text-only methods, including Hidden Markov Models (HMMs) and standard Linear-Chain Conditional Random Fields (L-CRFs), often struggle with long-range dependencies. For instance, identifying a question mark at the end of a sentence often requires "remembering" an interrogative word at the very beginning.

Methodology

The core contribution of this work is the application of Factorial Conditional Random Fields (F-CRFs), a type of Dynamic CRF, to the punctuation prediction task.

From Linear to Factorial CRFs

A standard L-CRF predicts a single sequence of tags based on local word context. While effective, it lacks a global view of sentence structure. The proposed F-CRF introduces a multi-layer architecture to capture these broader dependencies.

Graphical representation of the Factorial CRF (F-CRF) model showing joint modeling of word-level and sentence-level states.

As shown in the figure above, the F-CRF consists of two interconnected layers: the Word Layer ($y$), which predicts specific punctuation symbols (e.g., COMMA, PERIOD, QMARK) following each word, and the Sentence Layer ($z$), which predicts the underlying state of the sentence (e.g., "Start of Question"). By interconnecting these layers, the model performs joint inference, allowing global sentence-level information to guide local punctuation decisions.

Experimental Setup

The models were evaluated using the IWSLT09 corpus, featuring conversational speech in English and Chinese. Two subsets were utilized: BTEC (short tourism-related sentences) and CT (longer, more complex human-mediated dialogs). The task involves mapping a sequence of words to a sequence of punctuation tags, as illustrated below:

Example of punctuation tagging for a conversational speech transcript.

Key Results and Insights

The F-CRF model consistently outperformed HMM-based "Hidden Event" models and standard L-CRFs across all datasets.

  • Performance Gains: On the BTEC English dataset, the F-CRF achieved an F1-score of 88.56, significantly higher than the L-CRF (84.13). These gains persisted even on noisy ASR outputs, where the F-CRF reached an F1 of 93.19.
  • Structural Modeling: The results confirm that punctuation is a structural property of sentences. Jointly modeling sentence type and punctuation marks allows the model to capture dependencies that local windows miss.
  • Downstream Impact: Improved punctuation prediction directly benefited Machine Translation. Using F-CRF punctuation improved English-to-Chinese translation scores from 23.44 to 24.18 BLEU, a statistically significant improvement.

Conclusion

This research demonstrates that Dynamic CRFs, specifically Factorial CRFs, provide a robust framework for punctuation prediction in speech transcripts. By treating punctuation as a structural rather than purely local phenomenon, the F-CRF achieves state-of-the-art results. Future work may involve optimizing these models directly for downstream task performance, such as maximizing translation quality or parsing accuracy.

Summary updated

Abstract

This paper focuses on the task of inserting punctuation symbols into transcribed conversational speech texts, without relying on prosodic cues. We investigate limitations associated with previous methods, and propose a novel approach based on dynamic conditional random fields. Different from previous work, our proposed approach is designed to jointly perform both sentence boundary and sentence type prediction, and punctuation prediction on speech utterances. We performed evaluations on a transcribed conversational speech domain consisting of both English and Chinese texts. Empirical results show that our method outperforms an approach based on linear-chain conditional random fields and other previous approaches.
BibTeX
@inproceedings{lu2010better,
  author = {Wei Lu and Hwee Tou Ng},
  title = {{Better Punctuation Prediction with Dynamic Conditional Random Fields}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://openreview.net/forum?id=SJWxpZz_WB},
}
Figure