MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization

Gunjan Balde, Soumyadeep Roy, Mainack Mondal, Niloy Ganguly

Abstract

Pre-trained language models (PLMs) often struggle with specialized medical terminology due to a "tokenization tax," where complex terms are fragmented into meaningless sub-units. This paper introduces MEDVOC, a dynamic vocabulary adaptation strategy that optimizes vocabulary selection using a novel "fragment score." This approach achieves a 240x speedup in the adaptation process and significantly enhances the faithfulness and performance of medical text summarization across multiple benchmarks.

Introduction

General-domain Pre-trained Language Models (PLMs) like BART and PEGASUS are typically trained on corpora like Wikipedia and news articles. When applied to the medical domain, they encounter a "tokenization tax": specialized terms such as erythromycin or constipation are split into multiple meaningless sub-units (e.g., "er", "##yth", "##ro"). This fragmentation results in two primary issues: semantic loss, where the model fails to capture the concept's meaning, and unfaithful summaries, where the decoder incorrectly reassembles fragments into medically inaccurate text.

Methodology

MEDVOC addresses these challenges by treating the model's vocabulary as an optimizable parameter. The framework consists of several key components designed to bridge the gap between general-domain pre-training and medical-domain fine-tuning.

Dynamic Vocabulary Construction

Instead of using a fixed vocabulary, MEDVOC identifies "poorly tokenized" words in the target domain—specifically those split into more than three subwords—and adds them to the PLM's embedding matrix. This ensures that critical medical concepts are represented as single tokens.

Fragment Score Optimization

To determine the optimal set of words to add, the authors introduce the "fragment score," defined as the average number of subwords per word in reference summaries. Minimizing this score serves as a proxy for downstream task performance. This innovation allows for an efficient hyperparameter search, reducing the computational time required to find the "right" vocabulary from approximately 450 days to just 2 days.

Overview of the MEDVOC framework, illustrating the candidate subword selection and fragment score optimization process.

Targeted Intermediate Fine-Tuning

The authors utilize biomedical paper title generation (via the PubMed Abstracts Collection) as an intermediate task. This serves as a logical bridge to teach the model medical reasoning before it is fine-tuned on specific summarization datasets.

Experimental Results

The researchers evaluated MEDVOC across four diverse medical summarization benchmarks: BioASQ, EBM (Evidence-Based Medicine), MeQSum, and CHQSum (Consumer Health Questions).

Performance Gains

MEDVOC demonstrated significant improvements over standard PLM baselines:

  • Zero-Shot Performance: Outperformed baselines by 15.74% in ROUGE-L.
  • OOV Handling: In scenarios with high concentrations of Out-Of-Vocabulary (OOV) words, MEDVOC showed gains of 17.29%.
  • Efficiency: The strategy proved model-agnostic, showing consistent improvements across BertSumAbs, BART, and PEGASUS.
Comparative performance of MEDVOC against standard Intermediate Fine-Tuning (IFT-CNN) across different training scales and specific challenges like high OOV concentration.

Human Evaluation and Faithfulness

Beyond automated metrics, medical experts evaluated the generated summaries. MEDVOC summaries were rated as 88% faithful to the source text, compared to only 59% for standard baselines. By fixing the vocabulary bottleneck, the model no longer had to "hallucinate" the spelling of complex drug names or conditions.

Conclusion and Future Work

MEDVOC provides a computationally efficient and effective method for adapting PLMs to specialized domains. While it significantly improves faithfulness in single-document summarization, future work is needed to address multi-document summarization and deeper reasoning challenges. The authors plan to extend these vocabulary techniques to state-of-the-art Large Language Models (LLMs) to further enhance factual consistency in medical AI.

Summary updated

Abstract

This work presents a dynamic vocabulary adaptation strategy, MEDVOC, for fine-tuning pre-trained language models (PLMs) like BertSumAbs, BART, and PEGASUS for improved medical text summarization. In contrast to existing domain adaptation approaches in summarization, MEDVOC treats vocabulary as an optimizable parameter and optimizes the PLM vocabulary based on fragment score conditioned only on the downstream task's reference summaries. Unlike previous works on vocabulary adaptation (limited only to classification tasks), optimizing vocabulary based on summarization tasks requires an extremely costly intermediate fine-tuning step on large summarization datasets. To that end, our novel fragment score-based hyperparameter search very significantly reduces this fine-tuning time -- from 450 days to less than 2 days on average. Furthermore, while previous works on vocabulary adaptation are often primarily tied to single PLMs, MEDVOC is designed to be deployable across multiple PLMs (with varying model vocabulary sizes, pre-training objectives, and model sizes) -- bridging the limited vocabulary overlap between the biomedical literature domain and PLMs. MEDVOC outperforms baselines by 15.74% in terms of Rouge-L in zero-shot setting and shows gains of 17.29% in high Out-Of-Vocabulary (OOV) concentrations. Our human evaluation shows MEDVOC generates more faithful medical summaries (88% compared to 59% in baselines). We make the codebase publicly available at https://github.com/gb-kgp/MEDVOC.
BibTeX
@inproceedings{balde2024medvoc,
  author = {Gunjan Balde and Soumyadeep Roy and Mainack Mondal and Niloy Ganguly},
  title = {{MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure