Development and evaluation of automatic punctuation for French and English speech-to-text

Jáchym Kolář, Lori LamelInterspeech 2012

TL;DR

This paper tackles the crucial problem of automatically adding punctuation to speech-to-text (STT) transcripts, making them more readable and useful. They developed a system for French and English that combines acoustic (prosody) and textual (word N-grams) features using adaptive boosting, achieving significant improvements over text-only approaches.

What Problem Does This Paper Address?

Speech-to-text (STT) systems are great at transcribing spoken words, but the output is often a wall of text without punctuation. This makes it hard for humans to read and difficult for downstream natural language processing (NLP) tasks to understand. Adding punctuation automatically is key to bridging this gap. The challenge lies in accurately predicting where periods, commas, and question marks should go, especially when dealing with noisy, real-world speech data where STT systems have a significant word error rate. Previous work often focused on specific aspects or used cleaner data, leaving a need for a robust system that works under realistic conditions.

Methods

The core of the system is an adaptive boosting algorithm, specifically AdaBoost.MH, which combines multiple "weak" classifiers to create a strong one. This approach is flexible and can handle various feature types.

The system leverages two main sources of information:

  • Acoustic-Prosodic Features: These capture the rhythm and intonation of speech. Features include pause durations, vowel and final rhyme durations (to detect pre-boundary lengthening), and pitch (F0) and energy contours. These are extracted from the speech signal after forced alignment of the STT output.
  • Textual Features: These are primarily based on word N-grams (sequences of words) up to 4-grams, looking at words around the potential punctuation boundary. To compensate for the limited prosodic information available when STT errors occur, they also trained a large language model (LM) in an HMM framework and used its posteriors as additional features.

A two-pass classification strategy is employed: first, end-of-sentence marks (periods and question marks) are grouped, and then a second pass distinguishes between periods and question marks.

Datasets

The research uses approximately 50 hours of speech data in French and English from the Quaero project. This data includes Broadcast News (BN) and Broadcast Conversation (BC) genres, with BC being more interactive and spontaneous.

The data is split into:

  • Training set: ~417K-431K words.
  • Development set: ~42K-49K words (used for tuning features and model optimization).
  • Test set: ~34K-43K words.

Crucially, the paper highlights that they used two independent manual punctuation annotations for the test sets, which is a significant contribution for robust evaluation.

Key Insights and Ideas

  • Prosody Matters, But Text is King (Initially): While prosodic features are valuable, the system relying only on textual information performed significantly better than the prosody-only model. This suggests that word sequence is a strong indicator of punctuation.
  • The Power of Combination: The real gains come from combining textual and prosodic features. The "Text+Prosody" system consistently outperformed either feature set alone, showing that acoustic cues provide complementary information.
  • Robustness to STT Errors: The system is designed to work with imperfect STT output (word error rates of 17-19%). While performance degrades compared to using human-generated reference transcripts (REF), the combined system still provides useful punctuation. The relative increase in error due to STT errors was higher for French than English.
  • French vs. English Nuances: For human-annotated data (REF), the French system performed better, likely due to commas being easier to predict from text. However, under STT conditions, the performance for both languages became very similar, indicating that transcription errors diminish the advantage of text-based cues.
  • Question Marks are Tricky: Detecting question marks remains challenging, with relatively high error rates even for the combined system. This is attributed to their low frequency in the data, making robust training difficult.
  • Human Agreement as a Benchmark: Comparing system performance to inter-annotator agreement (using Cohen's Kappa) provides a realistic upper bound. The system's performance is lower than human agreement, highlighting the difficulty of the task in real-world scenarios.

Key Results

  • The Text+Prosody system achieved significantly better results than using either feature set alone.
  • Adding prosodic features reduced the Slot Error Rate (SER_LEN) by 11.0% to 19.6% relative to text-only systems, with higher gains on reference transcripts.
  • For French REF, the Text+Prosody system achieved an SER_LEN of 45.2%, compared to 56.6% for Text-only.
  • For English REF, the Text+Prosody system achieved an SER_LEN of 53.5%, compared to 63.4% for Text-only.
  • Under STT conditions, the Text+Prosody system achieved SER_LEN of 67.2% for French and 68.0% for English.
  • The relative increase in SER due to STT errors was 25.8% for English and 47.7% for French.
  • Human-automatic agreement (Kappa) was lower than inter-human agreement by 23.4% for French and 26.6% for English (on REF data).

Limitations and Future Work

  • STT Error Impact: The performance is still significantly impacted by STT word errors, especially for French.
  • Question Mark Detection: Accurately detecting question marks remains a challenge due to their sparsity.
  • Future Work: The authors plan to explore different modeling techniques and investigate the impact of automatic punctuation on transcript readability through perceptual tests. They also aim to address other languages.

Summary updated

BibTeX
@inproceedings{kolar2012development,
  author = {Jáchym Kolář and Lori Lamel},
  title = {{Development and evaluation of automatic punctuation for French and English speech-to-text}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure