Method for Designing Semantic Annotation of Sepsis Signs in Clinical Text

Melissa Yan, Lise Gustad, Lise Høvik, Øystein NytrøACL 2023

TL;DR

This paper introduces a systematic, iterative method for designing semantic annotation guidelines for clinical text, specifically focusing on sepsis signs. By grounding the process in clinical questions and continuously evaluating annotation quality through inter-annotator agreement, they demonstrate a way to build tailored, high-quality annotated corpora that are crucial for advancing clinical NLP research.

What Problem Does This Paper Address?

Annotated clinical text corpora are vital for training machine learning models to understand patient conditions, predict disease progression, and improve healthcare. However, the process of creating these corpora, particularly the design of annotation guidelines, is often underspecified, making it difficult to replicate, reuse, or adopt existing methods. This lack of transparency hinders the development of robust clinical NLP tools. The authors aim to fill this gap by detailing a practical, step-by-step methodology for designing effective semantic annotation guidelines.

Methods

The core of the paper is a novel, iterative method for designing annotation guidelines. This method is driven by:

  • Clinical Questions: The process begins by clarifying clinical questions (e.g., "Is there a connection between PIVCs and BSIs or phlebitis?"). These questions are then operationalized into specific corpus requirements, defining what information needs to be extracted.
  • Incremental Guideline Design: The annotation guideline is developed and refined incrementally. Each change to the guideline is tested by having annotators use it to annotate a corpus.
  • Continuous Evaluation: The quality of the annotations and the guideline itself are continuously evaluated using inter-annotator agreement (IAA). This feedback loop helps identify ambiguities, refine categories, and improve the guideline's clarity and comprehensiveness.
  • Synthetic Data Generation: To overcome data scarcity and privacy concerns, the authors generated synthetic clinical notes that mimic real-world data. This allowed for controlled experimentation and guideline testing.
  • Hierarchical Categories and Relationships: The annotation scheme uses hierarchical categories (e.g., Sign, Location, Device) with subcategories to capture detailed information. Relationships between these categories are also defined to capture contextual information.

Datasets

The study utilized a dataset of 18,555 Norwegian adverse event (AE) reports. To facilitate the annotation process and guideline development, the authors generated 100 unique synthetic clinical text notes. These synthetic notes were used for pre-annotation, guideline testing, and the main annotation phases.

Key Insights and Ideas

  • Clinical Questions as the North Star: Tying the entire annotation process to specific clinical questions ensures that the resulting corpus is directly relevant and useful for answering those questions, rather than being a generic collection of annotations.
  • Iterative Refinement is Crucial: A single pass at guideline design is insufficient. The iterative process of annotating, evaluating IAA, and revising the guideline is essential for uncovering ambiguities and improving clarity.
  • Synthetic Data is a Powerful Tool: Generating synthetic data can effectively address privacy concerns and data availability issues, enabling robust testing and development of annotation guidelines without compromising real patient data.
  • Granularity Matters for IAA: While finer granularity in annotation categories can capture more detail, it can also lead to lower inter-annotator agreement. The authors found that a balance is needed, and granularity can be adjusted based on downstream analysis needs.
  • Annotator Expertise Influences Results: The background and expertise of annotators (e.g., students vs. nurses vs. computer scientists) can significantly impact annotation consistency and the interpretation of guidelines.

Key Results

The paper details the systematic development of an annotation guideline for sepsis signs. While specific quantitative results on model performance are not the primary focus, the study demonstrates:

  • The successful creation of a structured annotation guideline with 7 main categories and hierarchical subcategories.
  • The use of synthetic data to generate a corpus of 770 annotated synthetic AE notes from 100 unique notes.
  • The iterative refinement of the guideline over 5 annotation sessions, leading to improved clarity and reduced ambiguity.
  • Insights into inter-annotator agreement (IAA) trends, showing how guideline complexity and annotator misuse of categories can affect agreement. For instance, they observed that the "main category" F1-score was often higher than the "as-is" score, indicating that while annotators might struggle with specific subcategories, they generally agree on the broader categories.

Limitations and Future Work

The authors acknowledge several limitations:

  • Specific Clinical Case: The method is demonstrated on a specific clinical case (sepsis signs) and may require adjustments for other clinical domains or problems.
  • Qualitative Focus: The study emphasizes the qualitative aspects of guideline design and annotation process.
  • Resource Dependency: The feasibility of the method, particularly the generation of synthetic notes and the number of annotators, depends on available resources.
  • Generalizability: Future work is needed to replicate this design and annotation process on other forms of clinical text and for different clinical problems to assess its broader applicability.

They suggest that future research should focus on replicating their method across diverse clinical contexts and exploring how to further optimize the balance between annotation granularity and annotator agreement.

Summary updated

BibTeX
@inproceedings{yan2023method,
  author = {Melissa Yan and Lise Gustad and Lise Høvik and Øystein Nytrø},
  title = {{Method for Designing Semantic Annotation of Sepsis Signs in Clinical Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure