Clinical Case Reports for NLP

Cyril Grouin, Natalia Grabar, Vincent Claveau, Thierry HamonACL 2019

TL;DR

This paper introduces a new, manually annotated French clinical case report corpus for Natural Language Processing (NLP) research. The corpus, comprising 717 reports with both general and fine-grained annotations, aims to facilitate the development of robust NLP tools for the biomedical domain, addressing the scarcity of French-language clinical data.

What Problem Does This Paper Address?

Accessing and utilizing textual data from specialized domains like medicine is crucial for developing effective NLP tools. However, privacy concerns and ethical considerations make clinical notes and similar documents difficult to obtain. While English-language clinical datasets like MIMIC-III exist, there's a significant lack of comparable resources for French. Existing French medical corpora are often limited in size or scope. Furthermore, de-identification techniques, while helpful, don't always guarantee complete removal of sensitive information and can impact linguistic specificity. This paper tackles the challenge of creating a valuable, accessible, and annotated French clinical corpus to bridge this gap and enable research in areas like information extraction and automatic classification.

Methods

The authors created a corpus of 717 French clinical case reports, which are publicly available. They manually annotated these reports at two levels:

  • General Information: Four categories were annotated: age, gender, outcome (e.g., recovery, improvement, death), and origin (the initial medical problem). Age, gender, and outcome values were normalized.
  • Fine-grained Information: A subset of 70 reports was further annotated with 27 fine-grained categories, covering domains like physiology, surgery, diseases, drugs, and temporal data. This involved defining specific annotation guidelines based on existing standards (UMLS, I2B2) and corpus-specific needs.

To assess the quality of their annotations, they computed inter-annotator agreement using Cohen's Kappa and F-measure. They also conducted basic experiments, including a linguistic analysis of verbal tenses and pronoun usage based on outcome, and an information extraction task using a linear chain Conditional Random Field (CRF) model on the fine-grained annotations.

Datasets

The primary dataset introduced in this paper is a corpus of 717 French clinical case reports. This corpus was manually annotated by domain experts. A subset of 70 reports was used for detailed fine-grained annotation and inter-annotator agreement analysis.

Key Insights and Ideas

  • The Value of Domain-Specific Corpora: The paper strongly emphasizes that building specialized corpora is essential for developing effective NLP tools, especially in domains like medicine where linguistic nuances are critical.
  • Bridging the French Language Gap: The creation of this French clinical corpus directly addresses a significant deficit in available resources for NLP research in this language.
  • Annotation Challenges: The authors highlight that certain categories, like "Outcome" and "Origin," are more challenging to annotate consistently due to subjectivity and the need for nuanced interpretation. This underscores the complexity of human annotation.
  • CRF Performance vs. Human Annotation: The study shows that while CRF models can achieve reasonable performance on information extraction tasks, they struggle with categories that are also difficult for human annotators (e.g., Concentration, Function, Pathology), suggesting that inherent data complexity is a key factor. Conversely, CRF models can sometimes outperform humans on categories with more regular patterns (e.g., Specialty, Dates).
  • Linguistic Patterns and Clinical Outcomes: The linguistic analysis revealed interesting correlations between verbal tenses and clinical outcomes, with past perfect being more common for death outcomes and present tense for stable or improved outcomes. This suggests that linguistic choices in case reports might reflect the perceived trajectory of a patient's condition.

Key Results

  • The corpus consists of 717 French clinical case reports.
  • General annotations: A total of 2,835 annotations were made across four general categories (age, gender, outcome, origin).
  • Fine-grained annotations: A subset of 70 files yielded 5,198 annotations across 27 fine-grained categories.
  • Inter-annotator agreement: While general information showed good agreement for Age and Gender (k=0.939), Outcome had poorer agreement (k=0.369), and Origin had very low agreement (k=-0.762). For fine-grained categories, the average F-measure was 0.5400.
  • Information Extraction (CRF): The CRF model achieved an overall F-measure of 0.67 on the 16 fine-grained categories, with Precision of 0.76 and Recall of 0.45.
  • Distribution of Annotations: The most frequent fine-grained categories were Tests (95.7%), Localisation (90.0%), and Signs or Symptoms (78.6%).

Limitations and Future Work

  • Limitations:
    • The inter-annotator agreement for "Outcome" and "Origin" was noted as problematic, indicating areas where annotation guidelines might need further refinement or where inherent ambiguity exists.
  • The CRF model's performance was lower for certain categories that were also difficult for human annotators, highlighting the challenges of extracting complex or irregular information.
  • The paper acknowledges that the current corpus is a subset of a larger, growing corpus, implying that the full potential is yet to be realized.
  • Future Work:
    • Continue fine-grained annotation of the entire corpus.
  • Define relationships between the existing annotated entities to create richer annotations.
  • Utilize the corpus for unsupervised experiments.
  • Develop automatic pre-annotation methods for relationships to streamline human annotation efforts.
  • The authors aim to make the corpus and its annotations publicly available to foster further research.

Summary updated

BibTeX
@inproceedings{grouin2019clinical,
  author = {Cyril Grouin and Natalia Grabar and Vincent Claveau and Thierry Hamon},
  title = {{Clinical Case Reports for NLP}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure