Analyzing the Granularity and Cost of Annotation in Clinical Sequence Labeling

Haozhan Sun, Chenchen Xu, Hanna SuominenarXiv 2021 · arXiv

Introduction

In supervised machine learning, high-quality, expert-annotated datasets are often considered the gold standard. However, in specialized domains such as medicine, obtaining these annotations is prohibitively expensive, requiring the time of highly trained professionals like registered nurses. This study investigates the "Return on Investment" (ROI) of manual labeling in clinical sequence labeling. Specifically, it asks whether increasing "annotation granularity"—providing more detailed, high-level clinical categories to guide a model—actually leads to better performance in identifying fine-grained sub-categories.

Methodology

The researchers developed a sequence labeling pipeline using a BiLSTM + CRF (Bidirectional Long Short-Term Memory + Conditional Random Field) architecture. To maximize model potential, they employed Stacked Embeddings, combining word-level (GloVe), contextual (BERT, ELMo, Flair), and document-level (Transformer) representations.

The study compared two primary scenarios:

  1. Baseline: A model utilizing only standard textual features (syntactic, semantic, and statistical data).
  2. Expert-Enhanced: A model incorporating additional manual annotations from an Australian registered nurse, which grouped tokens into high-level clinical categories.

Performance was evaluated using the Synthetic Nursing Handover Dataset, a high-quality set of 100 clinical records (approximately 16,217 tokens). The authors also introduced a quantitative ROI formula, calculating human labor costs at $0.08 per token against gains in the F1-score.

Key Results and Performance Analysis

The results indicate that adding specialized nursing knowledge provided negligible performance gains. Macro-averaged accuracy remained above 0.95 for both models, and macro-averaged Recall and F1-scores showed less than a 1% fluctuation regardless of the additional expert data.

Figure 5: Comparison of F-1 Scores before and after adding expert annotations across high-level clinical categories.

While there was a minor 2% increase in macro-averaged precision, the authors argue this is not substantial given the high cost of manual labor. For instance, the 'Medication' category saw a 5% precision boost, but other categories like 'NA' (tokens without significant clinical meaning) experienced a decrease in accuracy.

Figure 4: Precision comparison showing minor gains in specific categories like Medication after expert annotation.

The calculated ROI for the additional nursing annotation was remarkably low at 0.00023%. The study estimates that increasing the model's F1-score by just 1% through manual annotation would cost approximately $4,324.53 USD for a dataset of this size.

Discussion and Conclusion

The findings suggest that modern contextual embeddings (like BERT and ELMo) are already highly effective at capturing linguistic context, potentially making explicit expert labels redundant for certain clinical sequence labeling tasks. The authors advocate for a "frugal" approach to NLP development in specialized fields, prioritizing improved embeddings and efficient workflows over exhaustive, granular manual annotation.

Future research should explore whether these findings hold for more complex tasks like semantic parsing or if simpler, non-transformer models might still benefit from expert guidance. For now, the evidence suggests that the community may be over-investing in human labor where sophisticated machine representations already suffice.

Summary updated

Abstract

Well-annotated datasets, as shown in recent top studies, are becoming more important for researchers than ever before in supervised machine learning (ML). However, the dataset annotation process and its related human labor costs remain overlooked. In this work, we analyze the relationship between the annotation granularity and ML performance in sequence labeling, using clinical records from nursing shift-change handover. We first study a model derived from textual language features alone, without additional information based on nursing knowledge. We find that this sequence tagger performs well in most categories under this granularity. Then, we further include the additional manual annotations by a nurse, and find the sequence tagging performance remaining nearly the same. Finally, we give a guideline and reference to the community arguing it is not necessary and even not recommended to annotate in detailed granularity because of a low Return on Investment. Therefore we recommend emphasizing other features, like textual knowledge, for researchers and practitioners as a cost-effective source for increasing the sequence labeling performance.
BibTeX
@misc{sun2021analyzing,
  author = {Haozhan Sun and Chenchen Xu and Hanna Suominen},
  title = {{Analyzing the Granularity and Cost of Annotation in Clinical Sequence Labeling}},
  year = {2021},
  eprint = {2108.09913},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.09913},
}
Figure