Abstract
Pre-trained language models (PLMs) often struggle with specialized medical terminology due to a "tokenization tax," where complex terms are fragmented into meaningless sub-units. This paper introduces MEDVOC, a dynamic vocabulary adaptation strategy that optimizes vocabulary selection using a novel "fragment score." This approach achieves a 240x speedup in the adaptation process and significantly enhances the faithfulness and performance of medical text summarization across multiple benchmarks.
Introduction
General-domain Pre-trained Language Models (PLMs) like BART and PEGASUS are typically trained on corpora like Wikipedia and news articles. When applied to the medical domain, they encounter a "tokenization tax": specialized terms such as erythromycin or constipation are split into multiple meaningless sub-units (e.g., "er", "##yth", "##ro"). This fragmentation results in two primary issues: semantic loss, where the model fails to capture the concept's meaning, and unfaithful summaries, where the decoder incorrectly reassembles fragments into medically inaccurate text.
Methodology
MEDVOC addresses these challenges by treating the model's vocabulary as an optimizable parameter. The framework consists of several key components designed to bridge the gap between general-domain pre-training and medical-domain fine-tuning.
Dynamic Vocabulary Construction
Instead of using a fixed vocabulary, MEDVOC identifies "poorly tokenized" words in the target domain—specifically those split into more than three subwords—and adds them to the PLM's embedding matrix. This ensures that critical medical concepts are represented as single tokens.
Fragment Score Optimization
To determine the optimal set of words to add, the authors introduce the "fragment score," defined as the average number of subwords per word in reference summaries. Minimizing this score serves as a proxy for downstream task performance. This innovation allows for an efficient hyperparameter search, reducing the computational time required to find the "right" vocabulary from approximately 450 days to just 2 days.
Targeted Intermediate Fine-Tuning
The authors utilize biomedical paper title generation (via the PubMed Abstracts Collection) as an intermediate task. This serves as a logical bridge to teach the model medical reasoning before it is fine-tuned on specific summarization datasets.
Experimental Results
The researchers evaluated MEDVOC across four diverse medical summarization benchmarks: BioASQ, EBM (Evidence-Based Medicine), MeQSum, and CHQSum (Consumer Health Questions).
Performance Gains
MEDVOC demonstrated significant improvements over standard PLM baselines:
- Zero-Shot Performance: Outperformed baselines by 15.74% in ROUGE-L.
- OOV Handling: In scenarios with high concentrations of Out-Of-Vocabulary (OOV) words, MEDVOC showed gains of 17.29%.
- Efficiency: The strategy proved model-agnostic, showing consistent improvements across BertSumAbs, BART, and PEGASUS.
Human Evaluation and Faithfulness
Beyond automated metrics, medical experts evaluated the generated summaries. MEDVOC summaries were rated as 88% faithful to the source text, compared to only 59% for standard baselines. By fixing the vocabulary bottleneck, the model no longer had to "hallucinate" the spelling of complex drug names or conditions.
Conclusion and Future Work
MEDVOC provides a computationally efficient and effective method for adapting PLMs to specialized domains. While it significantly improves faithfulness in single-document summarization, future work is needed to address multi-document summarization and deeper reasoning challenges. The authors plan to extend these vocabulary techniques to state-of-the-art Large Language Models (LLMs) to further enhance factual consistency in medical AI.
Summary updated