TL;DR
This paper introduces BioNART, a novel non-auto-regressive transformer model for biomedical natural language generation. By adapting existing BERT architectures and leveraging pre-trained biomedical language models, BioNART achieves significantly faster inference speeds (up to 18x) for tasks like summarization and dialogue generation, while maintaining competitive performance.
What Problem Does This Paper Address?
Generating coherent and accurate text in specialized domains like biomedicine is crucial for applications such as clinical note summarization, patient-doctor dialogue generation, and medical question answering. Existing state-of-the-art generative models often rely on auto-regressive (AR) decoding, where text is generated token by token. While AR models produce high-quality output, their sequential nature makes them slow and computationally expensive during inference. This is a significant bottleneck for real-time applications or when processing large volumes of biomedical text. The paper aims to bridge this gap by exploring non-auto-regressive (NAR) generation in the biomedical domain, which promises faster inference by predicting all output tokens simultaneously.
Methods
The core innovation of BioNART lies in its adaptation of the non-auto-regressive (NAR) paradigm to the biomedical domain. BioNART is built upon an encoder-decoder architecture, similar to models like BERT2BERT.
- Encoder-Decoder Architecture: BioNART utilizes a standard encoder-decoder structure. The encoder processes the input text, and the decoder generates the output text.
- BERT Compatibility: Crucially, both the encoder and decoder are designed to be compatible with the widely used BERT architecture. This allows BioNART to leverage existing, powerful pre-trained biomedical language models (PLMs) like BioBERT and SciBERT for parameter initialization. This initialization significantly boosts performance and reduces the need for extensive training from scratch.
- Non-AutoRegressive Decoding: Unlike AR models, BioNART's decoder does not rely on previously generated tokens to predict the next one. Instead, it aims to predict all target tokens simultaneously.
- Connectionist Temporal Classification (CTC) Loss: To handle the challenges of NAR decoding, particularly aligning predicted tokens with their correct positions, BioNART employs the Connectionist Temporal Classification (CTC) loss. CTC is known for its flexibility in handling variable-length predictions and can efficiently marginalize over possible alignments.
- Additional Pre-training: To further adapt the pre-trained checkpoints for NAR decoding, the authors perform an additional pre-training step. This involves a permutation language modeling strategy, similar to XLNet, on biomedical abstracts from PubMed/MEDLINE. This helps the model learn to rearrange tokens and better fit the NAR objective.
Datasets
BioNART was trained and evaluated on several biomedical datasets:
- PubMed/MEDLINE Abstracts: Used for the additional pre-training phase.
- iCliniq: A dataset for biomedical text summarization, extracted from the MedDialog dataset. It contains patient-doctor dialogues and summaries.
- HealthCareMagic: Another dataset for biomedical text summarization, also extracted from MedDialog. It features more abstractive summaries written in a formal style.
- CovidDialog: A dataset for clinical dialogue generation, focusing on doctor-patient communication related to COVID-19.
Key Insights and Ideas
- NAR is Viable for Biomedicine: The paper demonstrates that non-auto-regressive generation, previously explored mainly in general NLP tasks, is a feasible and effective approach for biomedical text generation.
- Leveraging Existing PLMs is Key: The compatibility with BERT architectures and the ability to initialize with pre-trained biomedical PLMs (like BioBERT) are critical for BioNART's success, enabling strong performance with less training.
- CTC Loss for NAR Alignment: The use of CTC loss is a smart choice for NAR models, as it addresses the inherent challenge of predicting token positions without sequential dependencies.
- Additional Pre-training Boosts NAR: The authors show that an extra pre-training step specifically designed for NAR decoding further improves BioNART's performance, indicating that standard PLM pre-training might not be fully optimized for NAR objectives.
Key Results
- Speed: BioNART achieves an impressive 18x faster inference speed compared to auto-regressive models on the iCliniq dataset.
- Summarization Performance: On the iCliniq dataset, BioNART with additional pre-training achieved approximately 94% of the ROUGE-L score of the pre-trained auto-regressive BioBART model. For HealthCareMagic, it reached about 85% of BioBART's performance.
- Dialogue Performance: While BioNART showed a more significant performance gap on the CovidDialog dataset compared to AR models (likely due to its smaller size), the pre-trained version still demonstrated improved results.
- Parameter Initialization: Using BioBERT checkpoints for initialization significantly improved BioNART's ROUGE scores compared to random initialization.
- Environmental Impact: The paper notes that BioNART can achieve a reduction in CO2 emissions during inference compared to AR models.
Limitations and Future Work
- Limited Additional Pre-training: The authors acknowledge that the additional pre-training performed in this study was relatively small. They suggest further investigation into the optimal training steps, corpus size, and exploring self-training tasks beyond permutation language modeling.
- Performance Gap: While BioNART offers significant speedups, there is still a performance gap compared to the best AR models, particularly on smaller datasets like CovidDialog.
- Future Directions: The authors plan to extend BioNART to other downstream tasks such as relation extraction and entity linking, aiming to create a versatile and high-speed model for the biomedical domain.
Summary updated