Abstract
Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.
Introduction: The "Final Boss" of Speech Recognition
While ASR has matured in consumer electronics, the medical domain remains a significant challenge. Medical terminology is dense, nuanced, and frequently contains phonetically similar terms that confuse general-purpose models. A radiologist dictating "lobar collapse" might find their transcript reading "lower collapse"—a distinction with major clinical implications.
The primary obstacle to specialized medical STT is the lack of massive, high-quality paired audio-text datasets. This research bypasses the need for new STT engines by introducing a post-processing correction framework that leverages the visual context already available to clinicians: the medical images themselves.
The Multi-modal Medical Speech Module (MMSM)
The proposed solution is a post-processing framework that "sanity checks" the output of existing STT systems (like Google STT or Baidu DeepSpeech) against visual evidence.
The core of the system is the Multi-modal Medical Speech Module (MMSM). It utilizes a Vision-Language Pre-training (VLP) architecture based on the X-VL model. The process involves:
- Visual Encoding: A Vision Transformer (ViT-S/16) processes the Chest X-Ray (CXR) image.
- Textual Encoding: A 12-layer transformer processes the initial, potentially erroneous STT transcript.
- Cross-Attention Fusion: The model uses cross-attention layers to fuse image and text embeddings, allowing the system to verify if transcribed findings (e.g., "opacification") are supported by visual evidence in the image.
- Correction: A multi-modal decoder generates the final, corrected clinical report.
Training Strategy: Learning from "Corrupted" Data
To overcome the scarcity of paired audio-image-text data, the researchers employed a "corruption" training strategy. Instead of relying on real audio errors, they simulated them by taking existing medical reports from the MIMIC-CXR dataset and applying random removals, replacements, and insertions of words.
The model was trained to reconstruct the original, clean report using only the corrupted text and the corresponding CXR image. This forced the model to rely on visual context to "fill in the blanks" or correct nonsensical substitutions, such as misinterpreting "lymphangitic" as "Lynn fandrich."
Key Results and Clinical Impact
The MMSM was evaluated against six different STT systems and consistently showed improvements across both quantitative and clinical metrics.
- Quantitative Performance: The module significantly reduced Word Error Rate (WER). For instance, using the Coqui engine, the WER dropped from 0.446 to 0.139.
- Clinical Entity Accuracy: Using the RadGraph F1 metric, which evaluates the correctness of clinical entities and their relationships, the MMSM outperformed text-only baselines. This confirms that visual grounding helps preserve the clinical meaning of the transcription.
- Human Validation: A blind assessment by a board-certified radiologist gave the MMSM a score of 4.57/5.0, compared to 4.35 for text-only correction. The multi-modal approach was particularly effective at identifying "critical findings" that text-only models might miss or misinterpret.
Conclusion
Grounding medical speech in visual context provides a powerful anchor for transcription accuracy. By using images to disambiguate phonetically similar medical terms, the MMSM offers a path toward reliable, automated clinical documentation even in noisy environments. Future work will explore extending this multi-modal approach to more complex imaging modalities like CT and MRI, as well as integrating visual context directly into the ASR decoding process.
Summary updated