VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain

Khai Le-DucCOLING 2024

TL;DR

This paper introduces VietMed, the largest publicly available Vietnamese medical speech recognition dataset to date, addressing a critical data scarcity in this domain. They also release novel pre-trained and fine-tuned models that significantly outperform existing benchmarks, demonstrating the effectiveness of their approach even without in-domain pre-training.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems thrive on vast amounts of data, but the medical domain, especially for less-resourced languages like Vietnamese, suffers from a significant lack of publicly available speech datasets. This scarcity hinders research and development, making it difficult to build accurate ASR systems for medical applications. Existing medical ASR datasets are often small, limited in scope (e.g., focusing on specific diseases or accents), or lack publicly released models, making reproducibility a challenge. Furthermore, general-purpose Vietnamese ASR models often struggle with the specialized vocabulary and nuances of medical conversations.

Methods

The authors tackle this problem by creating a comprehensive dataset and developing advanced ASR models.

  • Dataset Creation: They collected a large corpus of Vietnamese speech, including:
  • VietMed-L: 16 hours of labeled medical speech.
  • VietMed-U: 1000 hours of unlabeled medical speech.
  • Viet-U: 1200 hours of unlabeled general-domain speech. The labeled data was meticulously annotated using a computer-assisted workflow to ensure high quality and efficiency. They also ensured broad coverage across ICD-10 disease groups and all Vietnamese accents.

  • Model Development: They focused on leveraging pre-trained models and fine-tuning them for the medical domain.

  • Pre-training: They developed w2v2-Viet and XLSR-53-Viet models. w2v2-Viet was trained from scratch on their general-domain data (Viet-U), while XLSR-53-Viet used the multilingual XLSR-53 model as initialization and was then pre-trained on Viet-U.
  • Fine-tuning: These pre-trained models were then fine-tuned on the labeled medical data (VietMed-Train). They also experimented with language models built using different lexicons.

Datasets

  • Newly Introduced:
    • VietMed: A comprehensive Vietnamese medical speech dataset comprising:
  • VietMed-L: 16 hours of labeled medical speech (split into train, dev, test).
  • VietMed-U: 1000 hours of unlabeled medical speech.
  • Viet-U: 1200 hours of unlabeled general-domain speech. This dataset is notable for its size, coverage of all ICD-10 disease groups, and inclusion of all Vietnamese accents.

  • Existing Datasets Used for Pre-training/Language Modeling:

    • XLSR-53: A multilingual speech representation model.
  • Various text corpora for language model training, including VIVOS, BABEL, CommonVoice, FOSD, VNTC-Health, VLSP 2020, ViHealthBERT-FAQ, and PhoNER-Covid19.

Key Insights and Ideas

  • Data Scarcity is a Major Bottleneck: The paper strongly emphasizes how the lack of public medical speech data, especially for languages like Vietnamese, is a significant barrier to progress in medical ASR.
  • Comprehensive Coverage Matters: By including all ICD-10 disease groups and all regional accents, VietMed aims to improve the generalizability of ASR models beyond narrow use cases.
  • Pre-training is Key for Low-Resource Domains: Even without using any in-domain medical data during the unsupervised pre-training phase, models initialized with large pre-trained representations (like XLSR-53) and then pre-trained on general Vietnamese data (Viet-U) show remarkable performance when fine-tuned on medical data.
  • Computer-Assisted Annotation is Efficient: The authors found that using a native speaker with a Biomedical Engineering degree to correct automatically generated transcripts reduced annotation time by 70% while improving quality.

Key Results

  • VietMed is the Largest: The paper claims VietMed is the world's largest public medical speech recognition dataset across several metrics (duration, speakers, diseases, etc.) and the largest public Vietnamese speech dataset by duration.
  • State-of-the-Art Performance: Their XLSR-53-Viet model, fine-tuned on VietMed-Train, achieved a Word Error Rate (WER) of 29.6% on the test set. This is a significant improvement over the baseline XLSR-53 model, which achieved 51.8% WER, representing a relative reduction of over 40%.
  • Competitive Models: Their w2v2-Viet model, pre-trained from scratch on general Vietnamese data, was competitive with XLSR-53 despite using significantly less data for pre-training.
  • Language Model Improvements: Augmenting the lexicon with their VietMed-Train data improved language model perplexity (PPL).

Limitations and Future Work

  • Unsupervised Pre-training on In-Domain Data: The authors acknowledge that they did not adapt their unsupervised pre-training models (w2v2-Viet, XLSR-53-Viet) using the unlabeled medical data (VietMed-U). They believe this could further enhance performance and leave it as future work.
  • Generalizability to Other Languages: While they aim to contribute to future research in medical ASR for other languages, the current work is focused on Vietnamese.
  • Further Exploration of Annotation: While their computer-assisted approach was effective, further research into optimal annotation strategies for medical speech could be beneficial.

Summary updated

BibTeX
@inproceedings{leduc2024vietmed,
  author = {Khai Le-Duc},
  title = {{VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure