TL;DR
This paper introduces the first manually annotated dataset for clinical Named Entity Recognition (NER) in Portuguese and demonstrates that training word embeddings on in-domain clinical text significantly improves NER performance compared to general-domain embeddings. The authors developed a BiLSTM-CRF model that achieves state-of-the-art results on this challenging task, paving the way for better information extraction from Portuguese clinical documents.
What Problem Does This Paper Address?
Extracting valuable information from clinical texts is crucial for healthcare professionals, enabling better decision-making and research. However, manually processing vast amounts of clinical data is impractical. Natural Language Processing (NLP) techniques, particularly Named Entity Recognition (NER), offer a solution by automatically identifying and classifying key entities like diseases, symptoms, and treatments. While NER research has advanced significantly, most work focuses on English. This paper addresses the critical gap of limited resources and research for clinical NER in Portuguese, a language with its own linguistic challenges. The authors aim to boost this field by creating a Portuguese clinical NER dataset, developing specialized word embeddings, and evaluating state-of-the-art models.
Methods
The core of this work involves building a robust NER system. The authors employed a BiLSTM-CRF (Bidirectional Long Short-Term Memory with a Conditional Random Field layer) neural network architecture, a common and effective choice for sequence labeling tasks like NER.
Key methodological contributions include:
- Dataset Creation: A new dataset of 281 Portuguese clinical case texts was manually annotated with specific named entities.
- In-Domain Word Embeddings: A novel set of word embeddings was trained specifically on a large corpus of Portuguese clinical texts (3,377 texts from the Sinapse journal). This contrasts with using pre-trained, general-domain embeddings. The FastText algorithm was used for training these embeddings, leveraging its ability to handle subword information.
- Model Training and Evaluation: The BiLSTM-CRF model was trained and evaluated using both the in-domain and a general-domain Portuguese word embedding model. The performance was compared using strict and relaxed F1-scores.
Datasets
- Training and Validation Dataset: 281 clinical case texts from the Portuguese journal Sinapse (volume 17).
- Testing Dataset: 20 clinical texts from the Neurology service of the Coimbra University Hospital Centre (CHUC), including admission notes, diagnostic test reports, and patient discharge letters.
- In-Domain Word Embedding Training Corpus: 3,377 clinical texts from all volumes of the Sinapse journal (2001-2018).
- Out-of-Domain Word Embedding Corpus: A general Portuguese word embedding model pre-trained on billions of tokens from Wikipedia and Common Crawl.
The newly introduced dataset is significant as it's the first manually annotated corpus for clinical NER in Portuguese, specifically designed to address the challenges of this language and domain.
Key Insights and Ideas
- In-Domain Embeddings Reign Supreme: The most striking finding is that word embeddings trained on a smaller, domain-specific corpus of Portuguese clinical texts significantly outperform general-domain embeddings. This highlights the importance of domain adaptation for NLP tasks, even when the in-domain data is less abundant.
- Clinical Language is Unique: The authors suggest that general-domain models struggle to capture specific clinical relationships (e.g., between diseases and diagnostic tests) that in-domain models learn effectively. This is evidenced by the differing "nearest neighbors" for medical terms in the two embedding types.
- BiLSTM-CRF is a Strong Performer: The BiLSTM-CRF architecture proves effective for Portuguese clinical NER, achieving state-of-the-art results on the newly created dataset.
- Annotation Challenges: The paper acknowledges that certain entity types, like "Additional Observations" and "Characterization," are more challenging to annotate consistently, leading to lower agreement ratios. This is attributed to their broad nature and potential overlap with other categories.
Key Results
- The in-domain WE model achieved higher performance (relaxed F1-score of 70.41%, strict F1-score of 62.71%) on the independent test set compared to the out-of-domain WE model.
- Validation results showed that the in-domain WE model consistently outperformed the out-of-domain model across various NE classes.
- The best hyperparameters for the BiLSTM-CRF model were found to be a hidden unit count of 26 and 25 for the in-domain and out-of-domain models, respectively, with a dropout percentage of 50% for both, suggesting that regularization is crucial for smaller datasets.
- The performance on the independent test set was slightly lower than on the validation set, potentially due to differences in text structure (admission notes vs. case reports) and the presence of orthographic errors in the test texts.
Limitations and Future Work
- Dataset Size: While a significant contribution, the dataset size (281 texts for training/validation) is relatively small, which can impact model generalization.
- Annotation Scope: The "Genetics" NE class was not present in the test set, limiting its evaluation. Similarly, "Negation" and "Route of Administration" had very few tokens in the test set.
- Model Architecture: The authors acknowledge that while BiLSTM-CRF is effective, exploring other deep learning architectures, such as those incorporating residual learning, could further improve performance.
- Parameter Optimization: The grid search for hyperparameters was limited due to time constraints, and further optimization could yield better results.
- Future Directions: The authors plan to conduct more extensive parameter optimization, explore advanced deep learning architectures, and expand the work to relation extraction between NEs to create a more comprehensive clinical NLP system. They also aim to make their datasets and models publicly available to foster further research.
Summary updated