TL;DR
This paper tackles the challenge of building robust German medical Named Entity Recognition (NER) models by harmonizing existing, disparate corpora into a unified meta-dataset. They demonstrate that while models trained on one corpus generalize poorly to others, combining multiple corpora significantly improves robustness, even across different medical fields.
What Problem Does This Paper Address?
The field of medical Natural Language Processing (NLP) has seen a surge in publicly available, semantically annotated German medical corpora. However, these corpora often suffer from differing data schemas (syntax) and entity definitions (semantics), making it difficult to combine them or train models that generalize well across them. This fragmentation hinders the creation of common meta-datasets and limits the practical utility of NER taggers trained on individual datasets. The core problem is understanding whether a model trained on one medical corpus can effectively identify entities in another, especially when those entities are defined differently or come from distinct medical domains.
Methods
The authors propose a multi-step approach:
- Harmonization: They leverage the BIGBIO framework to standardize the diverse data schemas of the German medical corpora into a common format. This involves creating custom parsers for each corpus to align their structure with BIGBIO's schema.
- Label Alignment: A crucial step is aligning the semantic entity classes across corpora. Since perfect equivalence is rare, they focus on entity classes that can be consistently mapped, particularly "medications." They explore different "span" configurations (short vs. long) to capture varying levels of detail in medication annotations.
- Meta-dataset Creation: By harmonizing and aligning labels, they create a unified meta-dataset.
- Cross-Corpus Evaluation: They fine-tune Transformer-based NER models on various combinations of training data from the meta-dataset and evaluate their performance on the test sets of individual corpora. This allows them to assess out-of-domain generalization.
- Transformer Checkpoint Comparison: They also compare the performance of different pre-trained Transformer models (general and biomedical-specific) on this cross-corpus task to understand their impact on robustness.
Datasets
The paper utilizes the following German medical corpora:
- BRONCO150: Oncology discharge summaries.
- GGPONC 2.0: Oncology clinical guidelines.
- GRASCCO: Synthetic case reports (annotated using GGPONC 2.0 guidelines).
- CARDIO:DE: Cardiology discharge summaries.
These corpora are harmonized into a unified meta-dataset using the BIGBIO framework. The paper does not introduce a new benchmark dataset but rather a harmonized meta-dataset derived from existing ones.
Key Insights and Ideas
- Poor Out-of-Domain Generalization: NER models trained on a single corpus perform significantly worse (around 20 percentage points drop in F1 score) when evaluated on unseen data from a different corpus, even within the medical domain.
- Benefit of Multi-Corpus Training: Combining multiple corpora for training, even if they come from different medical fields or text genres, substantially improves model robustness across various target datasets.
- Medications as a Consistent Target: Despite variations in annotation detail (e.g., dosage, strength), "medications" emerge as the most consistently mappable entity class across the studied corpora, making them a suitable focus for cross-corpus evaluation.
- No Clear Winner Among Transformers: Both general-purpose and biomedical-specific pre-trained Transformer models show similar performance in cross-corpus generalization, suggesting that the data harmonization and meta-dataset approach are more critical than the specific model architecture for this task.
- Non-Patient Data Can Help: Clinical guidelines (like GGPONC 2.0) can be beneficial for training models that are evaluated on patient-related data (like discharge summaries), highlighting the value of diverse data sources.
Key Results
- Performance Drop: On average, F1 scores dropped by approximately 20 percentage points when models were evaluated on unseen target corpora. The largest drops were observed for the GGPONC 2.0 test set (around 30 pp.).
- Improved Robustness: Training on all corpora combined (BRO+CAR+GGP+GRA) resulted in a significant increase in micro-averaged F1 scores (around 19 pp. for short-span and 18 pp. for long-span settings) compared to models trained on individual corpora, even if the performance on the "seen" training data was slightly lower.
- Transformer Performance: In out-of-domain settings, GELECTRA performed best in two out of three scenarios, while BioGottBERT and medBERT.de showed promising results, particularly in terms of producing fewer false negatives and false positives.
Limitations and Future Work
- Medication-Centric: The study is limited to "medication" entities due to the difficulty in aligning other semantic classes across all corpora.
- Subjective Alignment: The label alignment decisions, while based on thorough inspection, are acknowledged as somewhat subjective.
- Annotation Quality: The impact of varying annotation quality and potential biases across corpora is not fully explored, as measuring this is challenging.
- Hyperparameter Optimization: The study did not extensively explore hyperparameter optimization for each Transformer model, which could influence comparative results.
Future work suggested includes:
- Full Cross-Validation: Performing more comprehensive cross-validation experiments to obtain more stable results and confidence intervals.
- Exploring Other Label Alignments: Investigating other potentially comparable semantic classes (e.g., Diagnosis/Pathology, Therapeutic Procedure) as more aligned datasets become available.
- Generative Approaches: Exploring generative approaches for NER.
- Consensus on Standards: Advocating for more representative German clinical corpora and a consensus on semantics and annotation guidelines to facilitate inter-corpus operability.
Summary updated