Sentiment Reasoning for Healthcare

Khai Le-Duc, Khai-Nguyen Nguyen, Bach Phan Tat, Duy Le, Jerry Ngo, Long Vo-Dang, Anh Totti Nguyen, Truong-Son Hy

Abstract

Transparency in AI decision-making is crucial in healthcare due to the severe consequences of errors, and this is important for building trust among AI and users in sentiment analysis task. Incorporating reasoning capabilities helps Large Language Models (LLMs) understand human emotions within broader contexts, handle nuanced and ambiguous language, and infer underlying sentiments that may not be explicitly stated. In this work, we introduce a new task - Sentiment Reasoning - for both speech and text modalities, along with our proposed multimodal multitask framework and dataset. Our study showed that rationale-augmented training enhances model performance in sentiment classification across both human transcript and ASR settings. Also, we found that the generated rationales typically exhibit different vocabularies compared to human-generated rationales, but maintain similar semantics. All code, data (English-translated and Vietnamese) and models are published online: https://github.com/leduckhai/MultiMed

Introduction

In the high-stakes world of healthcare, "black-box" AI is a liability. If a model flags a patient’s tone as "Negative" or "Distressed," a clinician needs to know why to make an informed decision. Standard sentiment analysis often fails here because speech is subjective, complex, and heavily dependent on individual speaking styles. Existing research has largely focused on text-only sentiment or general-domain datasets. There is a significant gap in multimodal (speech + text) sentiment analysis specifically tailored for the medical domain, where transparency and trust are non-negotiable.

The Sentiment Reasoning Task

The authors propose a multimodal multitask framework for Sentiment Reasoning, which consists of two sub-tasks:

  1. Sentiment Classification: Labeling an input (human or ASR transcript) as POSITIVE, NEUTRAL, or NEGATIVE.
  2. Rationale Generation: Producing a free-form natural language explanation for that label.
The Sentiment Reasoning framework performing classification and rationale generation.

Methods

The researchers compared two main architectural approaches:

  • Cascaded Approach: An Automatic Speech Recognition (ASR) model transcribes audio to text, which is then fed into a Large Language Model (LLM).
  • End-to-End Spoken Language Models: Using models like Qwen2-Audio that process audio signals directly.

To train these models, they leveraged Rationale-Augmented Fine-Tuning. Instead of just training on labels, they used Chain-of-Thought (CoT) distillation, where models are trained on targets that include both the label and the human-annotated reason.

Datasets

The authors introduced the Sentiment Reasoning Dataset, built upon VietMed, a large-scale medical ASR corpus.

  • Scale: 30,000 samples, making it the largest of its kind.
  • Multilingual: Originally in Vietnamese, the transcripts were manually translated into English, Chinese (Simplified and Traditional), German, and French.
  • Content: Real-world doctor-patient conversations, annotated by professionals and verified by linguists.

Key Results and Insights

One of the most significant findings is the "Reasoning Bonus": adding the rationale generation task isn't just an add-on for humans—it actually helps the model learn better features. Rationale-augmented training led to a +2% increase in both accuracy and macro-F1 scores.

Performance comparison of various models on the Sentiment Reasoning task.
  • Domain Specificity: Domain-specific encoders (like ViHealthBERT) consistently outperformed general-purpose models.
  • Robustness to ASR Noise: Even with a Word Error Rate (WER) of nearly 30% from the ASR model, sentiment classification performance only dropped by about 5%, suggesting the sentiment "signal" is robust.
  • Semantic Similarity: Using BERTScore, the authors found that while AI-generated rationales use different vocabulary than humans, they are semantically very similar (scoring ~0.8).

Limitations and Future Work

The current "cascaded" system is complex to reproduce. While end-to-end models like Qwen2-Audio were tested, they are still catching up to the accuracy of cascaded systems in this domain. Future work will focus on integrating "paralinguistic" features—like tone and pitch—directly from audio and addressing the "Neutral" bias typical of medical consultations.

Summary updated

Abstract

Transparency in AI decision-making is crucial in healthcare due to the severe consequences of errors, and this is important for building trust among AI and users in sentiment analysis task. Incorporating reasoning capabilities helps Large Language Models (LLMs) understand human emotions within broader contexts, handle nuanced and ambiguous language, and infer underlying sentiments that may not be explicitly stated. In this work, we introduce a new task - Sentiment Reasoning - for both speech and text modalities, along with our proposed multimodal multitask framework and dataset. Our study showed that rationale-augmented training enhances model performance in sentiment classification across both human transcript and ASR settings. Also, we found that the generated rationales typically exhibit different vocabularies compared to human-generated rationales, but maintain similar semantics. All code, data (English-translated and Vietnamese) and models are published online: https://github.com/leduckhai/MultiMed
BibTeX
@inproceedings{leduc2024sentiment,
  author = {Khai Le-Duc and Khai-Nguyen Nguyen and Bach Phan Tat and Duy Le and Jerry Ngo and Long Vo-Dang and Anh Totti Nguyen and Truong-Son Hy},
  title = {{Sentiment Reasoning for Healthcare}},
  booktitle = {AIM-FM @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=WYn6YIjR41},
}
Figure