Abstract
Automatic Speech Recognition (ASR) systems often produce raw text streams lacking punctuation and sentence boundaries, which complicates human readability and degrades the performance of downstream Natural Language Processing (NLP) tasks. This paper introduces a joint prediction framework using Dynamic Conditional Random Fields (DCRF) to simultaneously identify sentence boundaries and insert punctuation marks. By integrating lexical data with prosodic cues and normalized language model scores, the proposed model achieves a 20% relative error reduction over traditional sequential methods. Experimental results on the TDT3 English broadcast news corpora demonstrate the robustness of this approach on both perfect transcripts and noisy ASR outputs.
Introduction
When a computer "listens" to speech via ASR, the output is typically a continuous stream of words without periods, commas, or capitalization. This "wall of text" is difficult for humans to parse and presents significant challenges for tasks like machine translation, which rely on clear sentence structures. Traditionally, researchers treated sentence boundary detection and punctuation insertion as two separate, sequential steps. However, these tasks are deeply intertwined; the type of sentence (e.g., declarative vs. question) directly dictates the appropriate punctuation. This paper addresses the challenge of joint prediction, aiming to improve accuracy by modeling these dependencies simultaneously.
Methodology
Model Architecture
The authors employ a Dynamic Conditional Random Field (DCRF), specifically a two-layer Factorial CRF (FCRF). Unlike standard linear-chain CRFs that predict a single label sequence, the FCRF utilizes two hidden layers: one for punctuation marks (Comma, Period, Question Mark, etc.) and another for sentence-level tags (e.g., "Beginning of a Question"). This architecture allows the model to capture the complex dependencies between sentence structure and specific punctuation marks.
Feature Engineering and Scaling
The model incorporates three primary feature types:
- Lexical: Standard n-grams (1-gram to 3-gram) within a 5-word sliding window.
- Prosodic: Acoustic cues derived from the raw audio, including pause duration, average pitch (F0) ratio, and energy ratio.
- Normalized N-gram Scores: A novel metric using language model probabilities to measure the likelihood of a punctuation mark versus a regular word in a given local context.
To manage the resulting "feature explosion" (often reaching millions of features), the researchers implemented vocabulary pruning—mapping rare words to a <UNK> token—and incremental local training to fit within hardware memory limits.
Experimental Results
The study utilized the TDT3 English broadcast news corpora, evaluating performance on both Correct Recognition Results (CRR) and actual ASR output (34.6% word error rate). The DCRF model consistently outperformed Linear-chain CRFs (LCRF), Maximum Entropy models (MaxEnt), and Hidden-event N-gram models (HEN).
Key Insights
- Joint Modeling Advantages: Predicting boundaries and punctuation simultaneously avoids "error propagation," where a mistake in boundary detection would inevitably lead to a punctuation error in a two-stage system.
- Resilience of Prosody: While lexical features are highly effective for perfect transcripts, they are sensitive to ASR errors. In contrast, prosodic features (derived directly from the audio) remain more resilient, making them vital for processing real-world ASR output.
- The Pruning Paradox: Surprisingly, pruning the vocabulary to the top 3,000 most frequent words improved the F1-measure. This reduction in data sparsity helped the model generalize better by treating rare, idiosyncratic words as a single "unknown" category.
Conclusion
The DCRF framework provides a significant advancement in enriching speech transcripts with structural metadata. While the model reduces prediction errors by 20%, a performance gap remains between perfect transcripts (61.4% F1) and ASR output (42.8% F1). Future work will explore tighter integration between the ASR engine and the punctuation model to mitigate alignment errors caused by word recognition failures.
Summary updated