Experiments on Sentence Boundary Detection

Mark Stevenson, Robert J. GaizauskasANLP 2000 · DOI · OpenReview

TL;DR

This paper tackles the challenge of automatically identifying sentence boundaries in transcriptions from Automatic Speech Recognition (ASR) systems, a crucial step for downstream NLP tasks. They establish a human baseline and propose a memory-based learning approach, finding that capitalization is a surprisingly important feature for this task, even more so than for standard text.

What Problem Does This Paper Address?

Most Natural Language Processing (NLP) research deals with well-punctuated, edited text like newspaper articles or novels. However, ASR systems produce raw transcriptions that are often in all caps, unpunctuated, and may contain errors. This "ASR text" lacks sentence boundaries, which are vital for many NLP applications such as part-of-speech tagging and parsing. Without sentence segmentation, linguistic analysis is severely limited. The paper aims to bridge this gap by exploring how to reliably detect sentence boundaries in this challenging ASR text format.

Methods

The authors first establish a human baseline by having subjects manually mark sentence boundaries in ASR text. They then implement a memory-based learning algorithm, specifically TiMBL, which categorizes word boundaries as either "sentence_boundary" or "no_boundary" based on similarity to learned examples. For feature engineering, they consider a range of linguistic cues around each word boundary, including preceding and following words, their part-of-speech tags, and whether they are stop words or capitalized. They experiment with and without capitalization features to assess its impact.

Datasets

  • British National Corpus (BNC): Mentioned as a source of 10 million words of manually marked sentence boundaries, useful for future corpus creation.
  • Wall Street Journal (WSJ): Used extensively for experiments. The WSJ text was pre-processed by removing punctuation and converting to all caps to simulate ASR text. A portion was used for training and a held-out portion for testing.
  • "The Nine O'Clock News" BBC television program: Transcriptions from two editions were used for the human annotation experiments.

Key Insights and Ideas

  • Capitalization is King (for ASR text): The most striking finding is that capitalization information significantly boosts performance for sentence boundary detection in ASR text. This is a key differentiator from standard text processing, where capitalization is often assumed or handled by other means.
  • ASR Text is Harder: The performance gap between processing standard text (even without punctuation) and ASR text (all caps, no punctuation) is substantial, suggesting that ASR transcriptions present unique challenges.
  • Human Agreement is Imperfect: While humans generally agree on sentence boundaries in standard text, their agreement can be lower in ASR text, especially when dealing with unpunctuated, all-caps input. The kappa statistic is introduced as a robust measure for assessing this agreement.
  • Context Matters (but how much?): While some prior work suggested minimal context is needed for punctuation disambiguation, this paper's results hint that more context might be beneficial for ASR text, especially if reliable part-of-speech tags are available.

Key Results

  • Human Performance: Human annotators showed a noticeable difference in performance between mixed-case (90-97% precision/recall) and upper-case (68-93% precision/recall) text, with mixed-case yielding higher results.
  • Algorithmic Performance: The memory-based system achieved a precision of 78% and recall of 75% on WSJ text with capitalization information. When capitalization was removed, performance dropped significantly to 36% precision and 35% recall.
  • Comparison to Random: Baseline random algorithms performed very poorly (around 5% precision/recall), highlighting the non-trivial nature of the task.
  • Kappa Statistic: High kappa values (0.91-0.98) were observed between human annotators, indicating strong agreement, though manual analysis revealed some disagreements.

Limitations and Future Work

  • Prosodic Information: The authors acknowledge that their current approach does not utilize prosodic information (like pause length, pitch) present in ASR output. They suggest that incorporating this could lead to improved results.
  • ASR Error Handling: The experiments were conducted on ASR text with punctuation removed but assumed to be error-free. Future work could explore handling actual transcription errors.
  • Generalizability: While the WSJ is a common benchmark, the authors aim to establish baselines for ASR text, suggesting further exploration is needed to generalize these findings to other types of ASR output.
  • Computational Approach: The paper focuses on highlighting the problem and establishing baselines rather than offering a definitive solution for ASR sentence boundary detection.

Summary updated

BibTeX
@inproceedings{stevenson2000experiments,
  author = {Mark Stevenson and Robert J. Gaizauskas},
  title = {{Experiments on Sentence Boundary Detection}},
  booktitle = {ANLP},
  year = {2000},
  doi = {10.3115/974147.974159},
}
Figure