FRASIMED: a Clinical French Annotated Resource Produced through Crosslingual BERT-Based Annotation Projection

Jamil Zaghir, Mina Bjelogrlic, Jean-Philippe Goldman, Soukaïna Aananou, Christophe Gaudet-Blavignac, Christian Lovis

TL;DR

Researchers have developed a novel pipeline to create large-scale clinical datasets for low-resource languages by "projecting" annotations from one language to another using Transformer models. The result is FRASIMED, now the largest open-access French clinical corpus with linked medical concepts, providing a vital resource for training medical Named Entity Recognition (NER) models.

What Problem Does This Paper Address?

In the world of Natural Language Processing (NLP), data is king. However, high-quality, human-annotated datasets are incredibly expensive and time-consuming to produce, especially in specialized fields like medicine. While English and Spanish have relatively robust clinical datasets, French has lagged behind.

This "data bottleneck" prevents the development of accurate clinical NER systems—tools that can automatically identify diseases, symptoms, and treatments in doctor's notes. The authors tackle this by asking: Can we take existing high-quality Spanish medical data and automatically translate both the text and its annotations into French with minimal human effort?

Methods

The authors propose a semi-automatic pipeline for crosslingual annotation projection. Instead of traditional string-matching, they leverage the power of BERT-based embeddings to ensure semantic accuracy.

The workflow, illustrated in the figure below, follows these key steps:

  1. Document-Level Translation: Spanish clinical cases are translated into French using DeepL Pro. The authors argue that document-level translation (rather than sentence-by-sentence) better preserves medical context.
  2. Sentence Alignment: They use LaBSE (Language Agnostic BERT Sentence Embedding) to map Spanish sentences to their French counterparts.
  3. Word Alignment: Using multilingual BERT and cosine similarity, the system identifies which specific words in the Spanish text correspond to words in the French translation. This is done bidirectionally to handle complex "one-to-many" mappings (e.g., one Spanish word becoming a multi-word phrase in French).
  4. Annotation Projection: The labels (like "Carcinoma") are transferred from the Spanish source to the French target.
  5. Quality Control: A semi-automatic report flags potential errors (like duplicate annotations or punctuation issues), followed by a final manual review using the Brat annotation tool.
The comprehensive workflow describing the semi-automatic methodology

Datasets

The project utilized two primary Spanish source datasets:

  • CANTEMIST: 1,301 synthetic clinical cases focused on oncology (cancer) and tumor morphology.
  • DISTEMIST: 750 cases annotated with various diseases linked to SNOMED-CT codes.

The resulting French dataset, FRASIMED, comprises 2,051 synthetic clinical cases. It is currently the largest open-source French corpus for oncological concepts, featuring over 24,000 annotations linked to ICD-O-3 and SNOMED-CT terminologies.

Key Insights and Ideas

  • Transformers Over String Matching: Previous attempts at annotation projection relied heavily on string matching or lemmatization, which often fail with the complex grammar and technical vocabulary of medical French. Using BERT embeddings allows the system to understand meaning rather than just spelling.
  • The "Gluttonous" Strategy: When the word aligner produced fragmented annotations (e.g., highlighting "tumor" and "malignant" but missing the word "and" in between), the authors used a "gluttonous" approach to fill the gaps, ensuring a cohesive and continuous annotation that favors recall.
  • Efficiency Gains: Creating these datasets manually would take months of expert labor. By using this projection method, the authors completed the manual correction of the entire CANTEMIST-FR dataset in just 28 hours.
  • Bidirectional Alignment is Crucial: As shown in Figure 4, aligning words in both directions (Spanish-to-French and French-to-Spanish) is necessary to capture the full scope of a medical entity when the translation changes the number of words used.
Bidirectional word alignment using cosine similarity

Key Results

The authors evaluated their automatic projection against a manually corrected "gold standard." * High Accuracy: For the CANTEMIST-FR dataset, the pipeline achieved a relaxed F1-score of 96.4%. Even under strict evaluation (requiring exact character-offset matches), it reached an 80.2% F1-score. * Consistency: The top 10 most frequent medical concepts in the new French dataset (like métastases and tumeur) perfectly mirrored the original Spanish data, proving that the semantic "soul" of the dataset remained intact through translation. * Resource Release: The full FRASIMED corpus has been made publicly available, providing a massive boost to the French clinical NLP community.

Limitations and Future Work

The primary limitation is the pipeline's dependence on the quality of the Machine Translation (MT) system. The authors noted that abbreviations and highly technical terms are still prone to mistranslation, which can lead to "bad projections" where the annotation is technically correct but the underlying French text is awkward or incorrect.

Future work could involve testing this pipeline on even lower-resource languages where high-quality MT might not be available, or exploring how to better handle the translation of idiosyncratic medical abbreviations.

Summary updated

Abstract

Natural language processing (NLP) applications such as named entity recognition (NER) for low-resource corpora do not benefit from recent advances in the development of large language models (LLMs) where there is still a need for larger annotated datasets. This research article introduces a methodology for generating translated versions of annotated datasets through crosslingual annotation projection. Leveraging a language agnostic BERT-based approach, it is an efficient solution to increase low-resource corpora with few human efforts and by only using already available open data resources. Quantitative and qualitative evaluations are often lacking when it comes to evaluating the quality and effectiveness of semi-automatic data generation strategies. The evaluation of our crosslingual annotation projection approach showed both effectiveness and high accuracy in the resulting dataset. As a practical application of this methodology, we present the creation of French Annotated Resource with Semantic Information for Medical Entities Detection (FRASIMED), an annotated corpus comprising 2'051 synthetic clinical cases in French. The corpus is now available for researchers and practitioners to develop and refine French natural language processing (NLP) applications in the clinical field (https://zenodo.org/record/8355629), making it the largest open annotated corpus with linked medical concepts in French.
BibTeX
@inproceedings{zaghir2024frasimed,
  author = {Jamil Zaghir and Mina Bjelogrlic and Jean-Philippe Goldman and Soukaïna Aananou and Christophe Gaudet-Blavignac and Christian Lovis},
  title = {{FRASIMED: a Clinical French Annotated Resource Produced through Crosslingual BERT-Based Annotation Projection}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure