New Arabic Medical Dataset for Diseases Classification

Jaafar Hammoud, Aleksandra Vatian, Natalia Dobrenko, Nikolai Vedernikov, Anatoly Shalyto, Natalia GusarovaarXiv 2021 · DOI · arXiv

TL;DR

This paper introduces a novel, specialized Arabic medical dataset for disease classification, addressing a critical gap in resources for Arabic NLP in the medical domain. They demonstrate its utility by fine-tuning state-of-the-art transformer models, showing superior performance compared to traditional methods.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen tremendous advancements, particularly with deep learning models, but these often rely on large, high-quality datasets. For the Arabic language, especially in specialized domains like medicine, such datasets are scarce. Existing Arabic datasets are often general-purpose, lacking the specificity needed for tasks like medical text classification. This deficiency hinders the development of AI tools that could significantly improve healthcare in Arabic-speaking regions, where electronic medical records are less common and often not digitally structured, or where English is used even in Arabic-speaking countries.

Methods

The authors propose a new Arabic medical dataset for text classification. They then evaluate its effectiveness by fine-tuning three pre-trained transformer-based models:

  • BERT (Bidirectional Encoder Representations from Transformers): A foundational model trained on a massive English corpus.
  • Arabert: A BERT-based model pre-trained on a large Arabic corpus, making it more suitable for Arabic language tasks.
  • ABioNER: Built upon Arabert, this model is further pre-trained on an Arabic biomedical literature corpus, aiming for even greater specialization.

These models are compared against two traditional "shallow" machine learning algorithms:

  • Support Vector Machines (SVM): A widely used supervised learning model for classification.
  • Naive Bayes (NB): A probabilistic classifier based on Bayes' theorem.

The paper details the fine-tuning process, including data splitting (80% train, 10% validation, 10% test), optimizer choice (Adam), learning rate, batch size, and sequence length. For the shallow algorithms, standard preprocessing steps like tokenization, stop-word removal, stemming, and TF-IDF vectorization were applied.

Datasets

The core contribution is the introduction of a new Arabic medical dataset.

  • Description: This dataset comprises 2,000 medical documents collected from various Arabic medical websites and the Arab Medical Encyclopedia.
  • Classes: It is categorized into 10 disease classes: Blood, Bone, Cardiovascular, Ear, Endocrine, Eye, Gastrointestinal, Immune, Liver, and Nephrological diseases.
  • Existing Datasets Mentioned: The paper also references existing Arabic corpora like KACST, BBC corpus, and CNN corpus, noting their general-purpose nature and lack of medical specialization.

Key Insights and Ideas

  • The critical need for domain-specific Arabic datasets: The paper strongly highlights that general-purpose datasets are insufficient for specialized NLP tasks in Arabic, particularly in the medical field.
  • Pre-trained models excel with domain adaptation: Fine-tuning large, pre-trained models (especially those with prior exposure to Arabic or medical text) on a specialized dataset yields significantly better results than traditional methods.
  • Specialization matters: The ABioNER model, pre-trained on both a general Arabic corpus and a specific Arabic medical corpus, achieved the highest accuracy, underscoring the benefit of multi-stage, domain-specific pre-training.
  • Transformer models outperform shallow algorithms: The experimental results clearly demonstrate the superiority of BERT, Arabert, and ABioNER over SVM and Naive Bayes for this medical text classification task.

Key Results

The paper presents F1 scores for the evaluated models. The ABioNER model achieved the highest F1 scores for both validation (97.4331) and testing (95.9124). Arabert v2 followed closely with validation (96.4327) and testing (94.5415) F1 scores. The original BERT model also performed well, with validation (94.1343) and testing (92.2934) F1 scores. In contrast, the shallow algorithms (SVM and NB) had significantly lower F1 scores, with SVM achieving 89.1308 (validation) and 87.3473 (testing), and NB achieving 87.6118 (validation) and 85.6949 (testing).

Limitations and Future Work

  • Limitations:
    • The dataset, while novel, is relatively small (2,000 documents) for deep learning standards, though it's a significant step for Arabic medical NLP.
  • The paper mentions that the dataset was collected semi-automatically, implying potential for noise or inconsistencies that were mitigated by expert review.
  • The authors note that shallow algorithms are sensitive to preprocessing, which they addressed, but the inherent limitations of these methods remain.
  • Future Work:
    • The authors state that this dataset is part of a larger project aiming to support other NLP tasks like Named Entity Recognition and Question-Answering systems.
  • They intend to make the dataset publicly available without preprocessing, allowing researchers to implement their own methods.
  • The success of specialized pre-trained models suggests future work could focus on further expanding and refining Arabic medical corpora for even more advanced models.

Summary updated

Abstract

The Arabic language suffers from a great shortage of datasets suitable for training deep learning models, and the existing ones include general non-specialized classifications. In this work, we introduce a new Arab medical dataset, which includes two thousand medical documents collected from several Arabic medical websites, in addition to the Arab Medical Encyclopedia. The dataset was built for the task of classifying texts and includes 10 classes (Blood, Bone, Cardiovascular, Ear, Endocrine, Eye, Gastrointestinal, Immune, Liver and Nephrological) diseases. Experiments on the dataset were performed by fine-tuning three pre-trained models: BERT from Google, Arabert that based on BERT with large Arabic corpus, and AraBioNER that based on Arabert with Arabic medical corpus.
BibTeX
@misc{hammoud2021new,
  author = {Jaafar Hammoud and Aleksandra Vatian and Natalia Dobrenko and Nikolai Vedernikov and Anatoly Shalyto and Natalia Gusarova},
  title = {{New Arabic Medical Dataset for Diseases Classification}},
  year = {2021},
  eprint = {2106.15236},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.15236},
}
Figure