Abstract
While large, biomedical documents with complex terminology are in need of being understood more easily and efficiently, summarizing this kind of content can be problematic, as Large Language Models (LLMs) aren't always trustworthy. Considering the importance of comprehending Cardiovascular Diseases, we study in depth the ability of different state-of-the-art biomedical LLMs to generate factual and certain summaries in this topic, and examine which generation choices can influence their trustworthiness. To that end, besides using factuality metrics, we employ techniques for token-level uncertainty estimation, an area that has received little attention from the scientific community. Our results reveal dissimilarities between LLMs and generation methods, and highlight connections between factuality and uncertainty metrics, thereby laying the groundwork for further investigation in the area.
Introduction
In the medical domain, the utility of Large Language Models (LLMs) for summarizing dense scientific literature is tempered by the risk of "hallucinations"—the generation of plausible-sounding but factually incorrect information. For cardiovascular disease (CVD) research, where clinical decisions hinge on precise data, these errors pose significant safety risks. This study addresses the "confident liar" syndrome, where models present inaccuracies with high internal certainty. Unlike previous work focusing on sequence-level confidence, this research investigates token-level uncertainty, aiming to identify local "flickers" of doubt that might signal factual errors in critical medical details like dosages or disease names.
Methodology
The researchers evaluated several open-source biomedical LLMs: BioMistral-7B, Meditron3-8B, Phi4-14B, and Qwen2.5 (7B and 14B). The evaluation framework utilized a dual-metric approach:
- Factuality Metrics: Hallucination Evaluation Model (HHEM) and AlignScore (measuring support of claims by the source text).
- Uncertainty Metrics: Token Certainty (probability of the selected token) and Token Entropy (the distribution of probability across the vocabulary).
The study utilized a specialized subset of the PubMed dataset, filtered via Medical Subject Headings (MeSH) for "Cardiovascular Diseases." The final dataset comprised 3,924 training records and 205 inference records, with original abstracts serving as the ground truth.
Key Insights and Results
The Confidence-Factuality Gap
A primary finding of this research is the lack of correlation between a model's internal confidence and the factual accuracy of its output. As shown in the correlation analysis, the relationship between Token Certainty and AlignScore is negligible.
The Pearson correlation ($r$) between certainty and factuality was approximately 0.0124, indicating that a model's self-reported confidence is a poor proxy for truthfulness in medical summarization.
Impact of Decoding Strategies
The choice of decoding strategy significantly influences both factuality and certainty. While "Greedy" decoding is often a default, the study found that Top-p sampling (nucleus sampling) yielded superior results.
For the Qwen-7B model, transitioning from Greedy to Top-p sampling reduced Token Entropy from 0.9868 to 0.1577—a substantial increase in model decisiveness and factual alignment.
Model Performance
The Qwen2.5 variants (7B and 14B) consistently outperformed other models, achieving the highest average ranks for both factuality and uncertainty. Interestingly, instruction fine-tuning (FT) did not provide a significant boost to factuality; instead, it often led to longer outputs that occasionally included information not present in the source text, potentially introducing new hallucinations.
Conclusion and Future Work
This study highlights the critical need for token-level uncertainty monitoring in medical AI. While models like Qwen2.5 show promise, the disconnect between confidence and factuality remains a major hurdle. Future research should expand to larger datasets and incorporate human expert evaluation by clinicians to determine the real-world safety of these automated summaries. Furthermore, benchmarking these open-source findings against proprietary models like GPT-4 will be essential for defining the state of the art in trustworthy biomedical AI.
Summary updated