AliBERT: A Pre-trained Language Model for French Biomedical Text

Aman Berhe, Guillaume Draznieks, Vincent Martenot, Valentin Masdeu, Lucas Davy, Jean-Daniel ZuckerACL 2023

TL;DR

This paper introduces AliBERT, a novel pre-trained language model specifically designed for French biomedical text. By developing a domain-specific tokenizer and training on a curated French biomedical corpus, AliBERT achieves state-of-the-art performance on downstream tasks, outperforming general-purpose French models.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen tremendous progress with the advent of large pre-trained language models (PLMs) like BERT. These models excel at understanding and generating human language by learning from massive text datasets. However, most existing PLMs are trained on general-domain text, and a significant portion of them are developed for English. While domain-specific PLMs (like BioBERT for English biomedical text) have shown great promise, there's a notable gap for non-English, domain-specific models. This is particularly true for French biomedical text, where existing general-purpose French PLMs (like CamemBERT and FlauBERT) struggle to capture the nuances of specialized terminology and context. The authors aim to bridge this gap by creating a French biomedical PLM that can better understand and process this specialized domain.

Methods

The core of this work is the development of AliBERT, a BERT-based language model. The authors introduce several key methodological contributions:

  • Domain-Specific Tokenization: Recognizing that biomedical text has unique vocabulary, they developed a custom Unigram sub-word tokenizer. This tokenizer was trained specifically on French biomedical text and incorporated a novel normalization step (adding spaces after punctuation) to improve vocabulary representation and reduce redundancy.
  • Pre-training Strategy: AliBERT was trained from scratch using a Masked Language Modeling (MLM) objective on a curated French biomedical corpus. They also explored an ELECTRA-based variant, AliBERT-ELECTRA, which uses a replaced token detection (RTD) mechanism.
  • Optimizer Exploration: The authors investigated the LAMB optimizer, known for faster training, and compared its performance and training time against the standard ADAM optimizer.

Datasets

The pre-training corpus for AliBERT was a significant undertaking, combining several sources of French biomedical text:

  • Drug leaflets: A public database of drug information.
  • RCP (Résumé des Caractéristiques du Produit): A French equivalent of the Physician's Desk Reference.
  • Biomedical articles: Collected from ScienceDirect.
  • Thesis manuscripts: From university documentation systems (SuDoc).
  • Cochrane databases: Articles from Cochrane.

This combined corpus amounted to approximately 7 GB of text.

For fine-tuning and evaluation, the authors used:

  • CAS dataset: A French biomedical dataset for Named Entity Recognition (NER), focusing on specific medical specialties.
  • QUAERO-MEDLINE dataset: Another NER dataset with UMLS semantic groups, used for more extensive experimentation.
  • Private dataset: A large dataset for biomedical text classification.

Key Insights and Ideas

  • Domain-Specific Tokenization is Crucial: The authors demonstrate that tailoring the tokenizer to the specific domain (biomedical French) significantly improves the model's ability to represent specialized vocabulary, which general-purpose tokenizers might miss.
  • Pre-training from Scratch Matters: Training a model from scratch on domain-specific data, rather than just fine-tuning a general model, leads to better performance on downstream domain tasks. This allows the model to learn domain-specific patterns and terminology more effectively.
  • Optimizer Choice Impacts Speed, Not Necessarily Performance: While LAMB offered faster training times, the ADAM optimizer ultimately yielded better performance on downstream tasks, highlighting a trade-off between speed and model quality.

Key Results

  • State-of-the-Art Performance: AliBERT and AliBERT-ELECTRA achieved state-of-the-art results on various French biomedical downstream tasks, including Named Entity Recognition (NER) and Masked Language Modeling (MLM).
  • Outperforming General Models: AliBERT consistently outperformed general-purpose French PLMs like CamemBERT on biomedical tasks, showing improvements in F1 scores for most entity types in NER. For instance, on the QUAERO dataset, AliBERT achieved a macro average F1 score improvement of around 15% over CamemBERT.
  • Faster Training with LAMB: The LAMB optimizer reduced training time significantly (e.g., from 4320 minutes to 76 minutes for BERT), though models trained with ADAM showed better performance.
  • Improved Accuracy in MLM: AliBERT showed a 23% increase in accuracy for masked word prediction compared to CamemBERT.

Limitations and Future Work

  • Corpus Diversity: The authors acknowledge that the biomedical corpus, while substantial, could benefit from greater diversity, particularly by including clinical documents like EHRs and physician notes.
  • Model Enlargement: They plan to integrate clinical data and explore continual learning strategies, potentially using existing French PLMs like CamemBERT as a base.
  • Longer Sequences: The current models are limited to 512 token sequences. Future work could explore models capable of handling longer sequences, like BigBird.
  • Text Generation: The team is also working on a new version of AliBERT with more data and plans to explore its use for generating biomedical texts.
  • Ethical Considerations: The paper briefly touches upon ethical considerations such as bias, privacy, and misinformation, emphasizing the need for responsible development and collaboration.

Summary updated

BibTeX
@inproceedings{berhe2023alibert,
  author = {Aman Berhe and Guillaume Draznieks and Vincent Martenot and Valentin Masdeu and Lucas Davy and Jean-Daniel Zucker},
  title = {{AliBERT: A Pre-trained Language Model for French Biomedical Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure