Abstract
Laboratory tests are crucial for diagnosing and managing health conditions, providing essential reference ranges for result interpretation. The diversity of lab tests, influenced by variables like the specimen type (e.g., blood, urine), gender, age-specific, and other influencing factors such as pregnancy, makes automated interpretation challenging. Automated clinical decision support systems attempting to interpret these values must account for such nuances to avoid misdiagnoses or incorrect clinical decisions. In this regard, we present LabQAR (Laboratory Question Answering with Reference Ranges), a manually curated dataset comprising 550 lab test reference ranges derived from authoritative medical sources, encompassing 363 unique lab tests and including multiple-choice questions with annotations on reference ranges, specimen types, and other factors impacting interpretation. We also assess the performance of several large language models (LLMs), including LLaMA 3.1, GatorTronGPT, GPT-3.5, GPT-4, and GPT-4o, in predicting reference ranges and classifying results as normal, low, or high. The findings indicate that GPT-4o outperforms other models, showcasing the potential of LLMs in clinical decision support.
Introduction
Interpreting a laboratory result is far more complex than simply checking if a numerical value falls within a generic range. A specific value might be considered healthy for a 25-year-old male but clinically concerning for a pregnant woman or an infant. Furthermore, reference ranges are highly sensitive to the specimen type (e.g., blood vs. urine) and the measurement units used (Traditional vs. SI).
Existing biomedical benchmarks often focus on general medical knowledge or document retrieval, frequently overlooking the "contextual nuance" required for precise lab data interpretation. This gap poses a risk for automated clinical decision support systems, where a lack of context can lead to misdiagnoses. To address this, researchers have introduced LabQAR, a dataset designed to evaluate an AI’s ability to retrieve correct reference ranges and classify results based on patient-specific variables.
Methodology
The development of LabQAR followed a rigorous four-stage pipeline to ensure clinical accuracy and data integrity:
- Data Selection: Reference ranges were sourced from authoritative medical texts, including Laposata’s Laboratory Medicine, the American Board of Internal Medicine, and Stanford Medicine.
- Data Extraction: Four expert annotators extracted data, mapping all tests to LOINC (Logical Observation Identifiers Names and Codes) and standardizing both traditional and SI units. Factors such as age, gender, and specimen type were meticulously recorded.
- Q&A Corpus Generation: Two sets of questions were developed. Set 1 consists of open-ended questions asking models to provide exact reference ranges given specific patient factors. Set 2 consists of multiple-choice questions where the model must classify a provided lab value as 'Normal', 'Low', or 'High'.
- Evaluation: The performance of several state-of-the-art LLMs was assessed using both exact match metrics and a margin-based assessment (±10% and ±20% of the bounds), reflecting the clinical reality where minor deviations may not always necessitate intervention.
The LabQAR Dataset
The primary contribution of this work is the LabQAR dataset, which includes:
- 550 lab test reference ranges.
- 363 unique lab tests.
- Specialized Subgroups: The dataset includes gender-specific (26), age-specific (7), pregnancy-related (15), and condition-specific (33) ranges to test model sensitivity to patient context.
Key Results and Insights
The benchmarking of several large language models revealed a significant performance hierarchy:
- GPT-4o Leadership: GPT-4o emerged as the top performer, achieving 63.3% accuracy on the classification task (Set 2). GPT-4 Turbo followed at 43.1%, while smaller models like LLaMA 3.1 (8B) struggled significantly, reaching only 26.7%.
- The Retrieval Challenge: Open-ended retrieval (Set 1) proved much more difficult. Even GPT-4o only achieved a 30.9% exact match rate for reference ranges. However, when the evaluation allowed for a 20% margin of error, its accuracy improved to 50.5%, suggesting that models possess a "fuzzy" understanding of boundaries rather than precise memorization.
- The "Nuance Gap": Models performed reliably on stable tests (e.g., Chloride) but faltered on complex tests like Follicle-stimulating hormone (FSH), which vary significantly based on physiological states like menopause.
- Domain Specialization: While GatorTronGPT (a model designed for the medical domain) outperformed the base LLaMA model, it still trailed the GPT-4 family, highlighting the importance of both model scale and high-quality training data.
Conclusion and Future Work
While GPT-4o shows promise, a 63.3% accuracy rate remains insufficient for safe clinical deployment, where misclassification can lead to dangerous medical errors. The study highlights a significant gap between current AI capabilities and the reliability required for "clinical-grade" decision support.
Future research will focus on implementing Retrieval-Augmented Generation (RAG) to allow models to pull real-time data from trusted sources like MedlinePlus, rather than relying on internal memory. Additionally, the researchers aim to test these models within actual Electronic Health Records (EHR) and use the LabQAR dataset to fine-tune smaller, open-source models to bridge the performance gap with proprietary systems.
Summary updated