SpecialtyScribe: Enhancing SOAP note Scribing for Medical Specialties using LLM’s

Sagar Goyal, Eti Rastogi, Fen Zhao, Dong Yuan, Andrew BeinsteinNAACL 2025

TL;DR

This paper introduces SpecialtyScribe, a modular framework for automatically generating high-quality medical SOAP notes, specifically tailored for specialized fields like oncology. By breaking down the process into information extraction, context retrieval, and note writing, SpecialtyScribe achieves performance comparable to proprietary models while being significantly smaller and more cost-effective.

What Problem Does This Paper Address?

The healthcare industry generates vast amounts of unstructured clinical data, with clinician-patient conversations forming the basis for crucial medical records. Generating accurate and concise SOAP (Subjective, Objective, Assessment, Plan) notes is vital for patient care and future treatment planning. However, creating these notes, especially in specialized fields like oncology, is challenging due to the need for high relevance, brevity, specificity, and adherence to clinician preferences. General-purpose Large Language Models (LLMs) often struggle with these requirements, and while newer LLMs show promise, their cost, size, and privacy concerns remain significant barriers for healthcare providers. Existing methods for medical note generation often fall short in real-world usability or fail to address the unique demands of specific medical specialties.

Methods

SpecialtyScribe employs a three-module pipeline:

  1. Information Extractor: This module is designed to capture oncology-specific, medically relevant information from the doctor-patient transcript. It's trained using an Orca-style instruction-following approach, where the model learns to extract information based on specific rules and sub-sections (e.g., Cancer Procedures, Cancer Symptoms).
  2. Context Retriever: This module aims to mitigate hallucinations and improve accuracy by verifying extracted information against the original transcript. It works by splitting the transcript into sentence chunks, embedding both the extracted information and transcript snippets, and then using embedding similarity to find relevant transcript context. A hallucination detection algorithm is also employed to further filter information.
  3. Note Writer: This module generates the final medical note using the filtered extracted information and contextual snippets. It's trained to produce subsection notes that combine to form the complete note, with the ability to ignore irrelevant information.

The framework leverages fine-tuned LLM-based models, specifically Mistral-7B, and incorporates techniques like Low-Rank Adaptation (LoRA) for efficient training.

Datasets

  • OncNoteGen Dataset: A proprietary dataset created by the authors, consisting of approximately 68,000 samples from 7,000 doctor-patient conversations, with an average context length of 7,000 tokens. This dataset was used for training the Information Extractor and Note Writer modules.
  • Aci-bench (subTask B): A public dataset used for benchmarking automatic medical visit note generation. 39 medical visits from this dataset were used for evaluation.
  • OncNoteGen Test: A proprietary test set of 21 oncology transcripts, designed to ensure coverage across various visit types, lengths, and styles. This dataset was used for both automated and human expert evaluation.

Key Insights and Ideas

  • Specialty-Specific Fine-Tuning is Crucial: The paper strongly suggests that generic LLMs, even advanced ones, are insufficient for high-quality specialty medical note generation. Fine-tuning on domain-specific data and tasks significantly improves performance.
  • Modular Approach Enhances Accuracy and Flexibility: Decomposing the note generation into distinct modules (Extraction, Retrieval, Writing) allows for better control, improved accuracy, and greater flexibility for adaptation to other specialties.
  • Context Retrieval is Key for Hallucination Mitigation: The Context Retriever module plays a vital role in grounding the generated notes in the actual transcript, significantly reducing the risk of factual inaccuracies or "hallucinations."
  • Cost-Effectiveness and Efficiency: SpecialtyScribe achieves performance comparable to much larger, proprietary models (like GPT-4 and Opus) while being a fraction of their size and cost, making advanced AI solutions more accessible to healthcare providers.

Key Results

  • Information Extraction: SpecialtyScribe's Information Extractor outperformed similarly sized open-source models by over 12% on ROUGE metrics and matched top closed-source LLMs while being less than 1% of their size. It showed particular strength in reducing "missed" entities.
  • Note Writing Quality: The Note Writer module, when evaluated using both reference-based metrics (BLEU, ROUGE) and human judgment, demonstrated superior performance on the proprietary OncNoteGen dataset compared to other models, highlighting the benefit of custom-trained models for specialized tasks.
  • Overall Medical Note Generation: Human experts rated SpecialtyScribe's generated notes highly for Writing Quality and Organization. While some proprietary models like GPT-40 showed better coverage, they struggled with note organization, an area where SpecialtyScribe excelled. The framework's performance was on par with or exceeded the latest open-source models and approached that of leading proprietary models.

Limitations and Future Work

  • Hallucination Mitigation: While the framework includes a hallucination detection mechanism, the authors suggest further work on specialized embedding models to improve detection and elimination of data hallucinations.
  • Generalizability to Other Specialties: The current work focuses on oncology. Future research should aim to extend the framework to include additional medical specialties to enhance its utility and robustness across various fields.
  • Open-Source Datasets: The authors note that open-source datasets may not always mirror real-world complexities, highlighting the need for more publicly available, representative datasets to drive progress.
  • Prompt Engineering: The paper mentions the potential for prompt leakage to influence outputs and suggests further investigation into prompt design and its impact.

Summary updated

BibTeX
@inproceedings{goyal2025specialtyscribe,
  author = {Sagar Goyal and Eti Rastogi and Fen Zhao and Dong Yuan and Andrew Beinstein},
  title = {{SpecialtyScribe: Enhancing SOAP note Scribing for Medical Specialties using LLM’s}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure