Sentence Detection Using Multiple Annotations

Ann Lee, James R. GlassInterspeech 2012 · DOI

Introduction

Automatic Speech Recognition (ASR) systems typically produce a continuous stream of words lacking punctuation. This absence of structure poses a significant challenge for downstream Natural Language Processing (NLP) tasks such as machine translation or parsing. Detecting sentence boundaries in spontaneous speech is particularly difficult due to the inherent subjectivity of punctuation and the limitations of local acoustic or linguistic cues. Researchers at MIT have proposed a robust framework that leverages multiple human annotations and a global sentence-length model to improve detection accuracy.

System Architecture and FST Framework

The core of the proposed system is a Finite-State Transducer (FST) framework that enables the fusion of diverse information sources into a single, globally optimized search. The system integrates three primary components: a prosodic model, language models, and a novel sentence length model.

System architecture for sentence boundary detection using FST composition.

The prosodic model utilizes an SVM classifier to analyze acoustic features such as pause duration and pitch. The language models include both standard n-gram models and syntactic models derived from CFG parsers. By representing these models as FSTs, the system can efficiently find the most probable path—and thus the most likely sentence boundaries—across all constraints simultaneously.

Modeling Global Sentence Length

A key innovation of this research is the inclusion of a global sentence-length model. Traditional systems often rely on local cues, which can be misleading in spontaneous speech. By modeling the typical number of words in a sentence using a Gamma distribution, the system can apply a "sanity check" to its predictions.

Distribution of words per sentence unit (SU) for sentence-level and clausal-level annotations.

As shown in the histograms above, sentence lengths in spontaneous speech follow a predictable statistical pattern. Integrating this distribution into the FST framework prevents the system from generating unnaturally short or long sentences, effectively "vetoing" boundaries that are statistically improbable for the genre.

Evaluation via Multiple Annotations

Recognizing that human agreement on punctuation is often low (with F-scores ranging from 0.48 to 0.64), the authors moved away from a single "gold standard" reference. Instead, they adapted the BLEU metric—commonly used in machine translation—to compare system output against ten different human annotations collected via crowdsourcing. This approach better accounts for the natural variability in how different people perceive sentence boundaries.

Results and Performance

The addition of the global sentence-length model yielded significant performance gains. On clean reference transcripts, detection performance improved by 7.2%. Even when applied to noisy ASR output with a 22% word error rate, the length model provided a 3.8% improvement. The system achieved a BLEU-like score of 0.56 on transcripts, nearing the human-to-human agreement baseline of 0.61.

Conclusion

By combining local acoustic features with global structural constraints and accounting for human subjectivity, this framework provides a more reliable method for punctuating spontaneous speech. While ASR errors remain a challenge for language and length models, the FST-based fusion approach offers a scalable path toward more readable and processable speech transcripts.

Summary updated

BibTeX
@inproceedings{lee2012sentence,
  author = {Ann Lee and James R. Glass},
  title = {{Sentence Detection Using Multiple Annotations}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-404},
}
Figure