TL;DR
This paper introduces a novel Named Entity Recognition (NER) model for automatically identifying COVID-19 symptoms from clinical conversations in Brazilian Portuguese. By leveraging transfer learning from an English-language biomedical NER model and a clever translation pipeline, they achieved competitive performance, enabling the tracking of symptom evolution and highlighting the need for enhanced mental health services.
What Problem Does This Paper Address?
The COVID-19 pandemic created an urgent need for automated tools to process vast amounts of clinical data. In Ceará State, Brazil, the "Plantão Coronavírus" (PC) chatbot was deployed to interact with citizens, generating a wealth of clinical conversation data. However, manually analyzing this data to track disease symptoms was infeasible. Furthermore, at the pandemic's outset, there was a significant lack of readily available NLP tools and models specifically for Brazilian Portuguese, especially for extracting medical entities like symptoms. This paper tackles the challenge of building an automated system to extract COVID-19 symptoms from these Portuguese-language clinical dialogues, enabling symptom tracking and analysis.
Methods
The core of this work is a symptom tracker built using a Named Entity Recognition (NER) model. The authors employed a transfer learning approach, a common technique where knowledge gained from one task is applied to a related but different task. Given the scarcity of Brazilian Portuguese NER models for medical contexts, they adapted an existing English-language biomedical NER model, ScispaCy (specifically the en_ner_bc5cdr_md model).
Their innovative pipeline involved:
- Collecting dialogues: Gathering clinical conversations from the Plantão Coronavírus chatbot.
- Translation to English: Translating the Portuguese dialogues into English using Google Translate.
- Symptom Extraction (English): Using the ScispaCy model to identify and extract symptoms from the translated English texts.
- Translation back to Portuguese: Translating the extracted English symptoms back into Brazilian Portuguese.
- Training Set Creation: Combining the original Portuguese dialogues with the translated Portuguese symptoms to create a training dataset.
- NER Model Training: Training a new NER model using spaCy, incorporating the created dataset.
This approach bypasses the need for manual annotation of the large Portuguese dataset, which would be time-consuming and expensive.
Datasets
The primary dataset used in this research is the Plantão Coronavírus dataset. This dataset consists of approximately 27,690 dialogues, containing 577,814 utterances, collected from the "Plantão Coronavírus" chatbot in Ceará State, Brazil. The data used for training and testing was specifically selected from April and May 2020 to ensure consistency in the COVID-19 protocol followed by the system during that period.
Key Insights and Ideas
- Bridging the Language Gap: The paper demonstrates a successful strategy for building NLP models for low-resource languages (like Brazilian Portuguese in this domain) by leveraging existing high-quality models from well-resourced languages (like English) through translation and transfer learning.
- Beyond Physical Symptoms: The model's ability to identify altered psychological behaviors (anxiety, anguish, sadness) as symptoms is a significant finding. This highlights the broader impact of the pandemic on mental health and the potential for such tools to inform public health policy.
- Automated Annotation Efficiency: The reliance on automated translation and an existing NER model for annotation significantly reduces the effort required to build a specialized NER model, making it a scalable approach.
- Tracking Symptom Evolution: The developed tracker can monitor the emergence and frequency of specific symptoms over time, offering valuable insights into the pandemic's progression and potential new disease manifestations.
Key Results
The authors report a competitive F1-score of 85.66% for their trained NER model on the test set. This performance is comparable to the English ScispaCy model's F1-score of 85.02%. The model successfully identified a high incidence of altered psychological behaviors, leading to recommendations for expanding mental health services in Ceará State.
Limitations and Future Work
- Limitations:
- The paper acknowledges that "Diabetes" was identified as a symptom, which they attribute to potential contextual factors or biases introduced by the ScispaCy model. While noting the association between hyperglycemia and COVID-19, they state this falls outside the paper's scope.
- The dataset was limited to two months (April-May 2020) to ensure protocol consistency, potentially limiting the analysis of symptom evolution over longer periods or during different pandemic phases.
- The use of Google Translate for text translation, while effective, can introduce noise, though the authors deem it insignificant for their work.
- Future Work:
- Extend the NER model to identify symptoms for other prevalent diseases in Brazil, such as influenza.
- Explore various neural architectures for improved performance.
- Investigate the translation performance more rigorously, potentially using metrics like BLEU scores with manually translated samples.
- Examine the evolution of "altered physiological behaviors" in dialogues from periods beyond April-May 2020.
- Consider enriching the model with BioBERTpt, a Portuguese-specific BERT model, and potentially experimenting with its performance.
Summary updated