TL;DR
This paper introduces PMP (Punctuation Mark Prediction), a small and fast BERT-based model for Chinese medical punctuation restoration. By combining knowledge distillation, supervised contrastive learning, and a novel auxiliary pre-training task, PMP achieves 95% of the performance of a much larger state-of-the-art model while using only 10% of its size, making it practical for real-world clinical applications.
What Problem Does This Paper Address?
In clinical settings, automatic speech recognition (ASR) systems often produce transcripts without punctuation. This lack of punctuation can lead to misinterpretations of medical reports, hindering clear communication and potentially impacting patient care. Automatic Punctuation Restoration (APR) aims to fix this by re-inserting punctuation. However, existing powerful models are often too large and slow for practical deployment in resource-constrained environments. The authors tackle the challenge of creating a highly performant yet computationally efficient APR model specifically for Chinese medical dictation.
Methods
The core of the proposed solution is a "pre-training and fine-tuning" paradigm.
- Pre-training:
- Knowledge Distillation (KD): To create a small model, they distill a larger "teacher" model (Chinese RoBERTa-wwm) into a smaller "student" model. This involves training the student to mimic the teacher's intermediate layer representations.
- Punctuation Mark Prediction (PMP): A novel auxiliary pre-training task is introduced. Similar to Masked Language Modeling (MLM), it masks tokens and trains the model to predict punctuation marks. This task is specifically designed to improve punctuation restoration capabilities.
- Supervised Contrastive Learning (SCL): To address the data imbalance issue (many more non-punctuation tokens than punctuation tokens), SCL is incorporated. It pulls similar punctuation instances closer in the latent space and pushes dissimilar ones apart, enhancing the model's ability to distinguish between punctuation types.
- The final pre-training loss is a combination of standard cross-entropy (CE) loss and SCL, weighted by a hyperparameter.
- Fine-tuning:
- Slot Tagging: To bridge the gap between the pre-training (which uses artificial
[MASK]tokens) and the real-world task (which has no[MASK]tokens), APR is reformulated as a slot tagging problem. Special[M]tokens are inserted between Chinese words, and the model predicts punctuation tags for these slots. This approach ensures that punctuation is not inserted within a single Chinese word.
- Slot Tagging: To bridge the gap between the pre-training (which uses artificial
Datasets
- Pre-training: The authors created a large pre-training dataset of 50 million tokens by crawling Chinese medical reports from online medical websites.
- Fine-tuning: A dataset named CMRPT (Chinese Medical Reports Punctuation Transcription) was curated from the crawled data. It was split into training, development, and testing sets. For fine-tuning, the focus was on three punctuation marks: comma (,), period (.), and colon (:), as exclamation and question marks were deemed less frequent and less critical for practical applications.
Key Insights and Ideas
- Specialized Pre-training for APR: The Punctuation Mark Prediction (PMP) task is a crucial innovation. Instead of relying solely on general language understanding from MLM, PMP directly trains the model to predict punctuation, making it more effective for the downstream task.
- Contrastive Learning for Imbalance: SCL proves effective in handling the severe data imbalance in punctuation restoration, a common challenge in sequence labeling tasks.
- Slot Tagging as a Bridge: Reformulating APR as slot tagging during fine-tuning is a clever way to align the model's output with the real-world task, overcoming the limitations of artificial tokens used during pre-training.
- Distillation for Efficiency: Knowledge distillation is shown to be a viable strategy for creating significantly smaller models that retain a high percentage of the performance of much larger, state-of-the-art models.
Key Results
- The proposed PMP model (specifically PMP-H256 and PMP-H312) achieves approximately 95% of the performance of the state-of-the-art Chinese RoBERTa-wwm model.
- Crucially, these PMP models have about 10% of the model size of RoBERTa-wwm.
- When compared to other distilled models with similar parameter counts (e.g., MiniRBT, TinyBERT), PMP models consistently achieve better performance.
- An ablation study confirms that the PMP auxiliary task, SCL, and KD all contribute positively to the overall performance. Removing KD led to a larger model size with similar performance, while removing SCL or CE resulted in performance degradation.
Limitations and Future Work
- Limited Punctuation Set: The fine-tuning dataset focuses on only three punctuation marks (comma, period, colon). The paper mentions that exclamation and question marks were excluded due to lower frequency, but their inclusion could be a future direction.
- Model Size vs. Performance Trade-off: While the paper demonstrates a significant reduction in model size, there's still a slight performance drop compared to the teacher model. Future work could explore further optimizations to close this gap.
- Generalizability: The model is trained and evaluated on Chinese medical reports. Its performance on other domains or languages is not discussed.
- Future Work: The authors suggest that their PMP model can be effectively fine-tuned for Chinese medical punctuation restoration, implying that further exploration of its application in this domain is warranted. The success of the proposed techniques could also be applied to other sequence labeling tasks with similar challenges.
Summary updated