Comparing Evaluation Metrics for Sentence Boundary Detection
Lists
Only you see this
Abstract
In recent NIST evaluations on sentence boundary detection, a single error metric was used to describe performance. Additional metrics, however, are available for such tasks, in which a word stream is partitioned into subunits. This paper compares alternative evaluation metrics - including the NIST error rate, classification error rate per word boundary, precision and recall, ROC curves, DET curves, precision-recall curves, and area under the curves - and discusses advantages and disadvantages of each. Unlike many studies in machine learning, we use real data for a real task. We find benefit from using curves in addition to a single metric. Furthermore, we find that data skew has an impact on metrics, and that differences among different system outputs are more visible in precision-recall curves. Results are expected to help us better understand evaluation metrics that should be generalizable to similar language processing tasks.
BibTeX
@inproceedings{liu2007comparing,
author = {Yang Liu and Elizabeth Shriberg},
title = {{Comparing Evaluation Metrics for Sentence Boundary Detection}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2007},
doi = {10.1109/icassp.2007.367194},
}Loading related content...