Abstract
Skin carcinoma is the most prevalent form of cancer globally, accounting for over $8 billion in annual healthcare expenditures. In clinical settings, physicians document patient visits using detailed SOAP (Subjective, Objective, Assessment, and Plan) notes. However, manually generating these notes is labor-intensive and contributes to clinician burnout. In this work, we propose a weakly supervised multimodal framework to generate clinically structured SOAP notes from limited inputs, including lesion images and sparse clinical text. Our approach reduces reliance on manual annotations, enabling scalable, clinically grounded documentation while alleviating clinician burden and reducing the need for large annotated data. Our method achieves performance comparable to GPT-4o, Claude, and DeepSeek Janus Pro across key clinical relevance metrics. To evaluate clinical quality, we introduce two novel metrics MedConceptEval and Clinical Coherence Score (CCS) which assess semantic alignment with expert medical concepts and input features, respectively.
Introduction
Clinical documentation is a primary driver of physician burnout across medical specialties. In dermatology, the standard for documenting patient encounters is the SOAP format: Subjective (patient history), Objective (physical exam findings), Assessment (diagnosis), and Plan (treatment strategy). While critical for continuity of care, the manual creation of these notes is time-consuming.
Existing automated solutions often rely on large-scale doctor-patient dialogue datasets, which are notably scarce in dermatology. In this field, diagnostic reasoning is frequently visual or based on brief clinical observations rather than lengthy transcripts. This research addresses the gap by proposing a framework that bridges visual lesion data and structured medical narratives without requiring massive, human-annotated datasets.
Methodology
The proposed framework utilizes a three-phase approach to move away from heavy reliance on expert human labels, focusing instead on weak supervision and data synthesis.
- Pre-training (Data Synthesis): To overcome the scarcity of expert-labeled SOAP notes, the researchers developed a "weakly supervised" dataset. Sparse clinical features (e.g., lesion diameter, biopsy status) were transformed into coherent captions using GPT-3.5. These captions served as queries for a Retrieval-Augmented Generation (RAG) system, which pulled context from authoritative medical databases like the National Cancer Institute. This enriched context was then used by a Vision-LLaMA 3.2 model to generate "pseudo-label" SOAP notes.
- Fine-Tuning: The Vision-LLaMA 3.2 model was fine-tuned on this synthesized dataset using Parameter-Efficient Fine-Tuning (PEFT) via QLoRA. This method allows for high-performance adaptation with significantly reduced computational overhead.
- Inference: During the final stage, the model accepts a new lesion image and a brief clinical caption to generate a complete, structured SOAP note.
Evaluation Metrics
To move beyond standard NLP metrics like ROUGE, which often fail to capture clinical nuance, the authors introduced two novel benchmarks:
- MedConceptEval: This metric measures how accurately generated notes align with expert-curated "descriptor banks" of medical concepts.
- Clinical Coherence Score (CCS): Utilizing ClinicalBERT embeddings, this score ensures the generated note remains internally consistent with the provided input features.
Results and Discussion
The framework demonstrated clinical fidelity comparable to state-of-the-art general-purpose models. Using an "LLM-as-a-Judge" framework, the model achieved a perfect 20/20 score across structure, readability, completeness, and medical relevance, outperforming models like GPT-4o and Claude 3.7.
Statistical analysis (ANOVA) indicated that the model successfully learned the "logic" of medical reporting, as the organization of information within the SOAP structure had a more significant impact on semantic quality than the specific skin condition being documented. The model showed particularly strong performance in the Assessment and Plan sections, where visual cues from the multimodal input are paramount.
Conclusion
This work demonstrates that weak supervision is a viable path for generating high-quality clinical documentation. By integrating visual features with sparse text through a RAG-enhanced pipeline, the framework achieves professional-grade SOAP note generation. Future work will focus on mitigating hallucination risks, expanding the framework to longitudinal patient data, and incorporating human-in-the-loop refinement strategies to ensure safety in real-world clinical deployments.
Summary updated