Comparing Evaluation Metrics for Sentence Boundary Detection

Yang Liu, Elizabeth Shriberg

Abstract

In recent NIST evaluations on sentence boundary detection, a single error metric was used to describe performance. Additional metrics, however, are available for such tasks, in which a word stream is partitioned into subunits. This paper compares alternative evaluation metrics - including the NIST error rate, classification error rate per word boundary, precision and recall, ROC curves, DET curves, precision-recall curves, and area under the curves - and discusses advantages and disadvantages of each. Unlike many studies in machine learning, we use real data for a real task. We find benefit from using curves in addition to a single metric. Furthermore, we find that data skew has an impact on metrics, and that differences among different system outputs are more visible in precision-recall curves. Results are expected to help us better understand evaluation metrics that should be generalizable to similar language processing tasks.
BibTeX
@inproceedings{liu2007comparing,
  author = {Yang Liu and Elizabeth Shriberg},
  title = {{Comparing Evaluation Metrics for Sentence Boundary Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2007},
  doi = {10.1109/icassp.2007.367194},
}

Loading related content...

Figure