Empirical Review of LLM-driven Classification of Multidimensional Sleep Health Mentions from Free-Text Clinical Notes

Syed-Amad Hussain, Ariana Calloway, Joseph Sirrianni, Eric Fosler‐Lussier, Mattina DavenportmedRxiv 2025 · DOI

Abstract

Accurate multidimensional sleep health (MSH) information is often fragmented and inconsistently represented within hospital infrastructures, leaving crucial details buried in unstructured clinical notes rather than discrete fields. This inconsistency complicates large-scale phenotyping, secondary analyses, and clinical decision support regarding sleep-related outcomes. In this work, we systematically explore contemporary natural language processing techniques, prompt-based large language models (LLMs) and fine-tuned discriminative classifiers, to bridge this critical gap. We evaluate performance on extracting nine key MSH dimensions (timing, duration, efficiency, sleep disorders, daytime sleepiness, interventions, medication, behavior, and satisfaction) from clinical narratives using public datasets (MIMIC-III derivatives) and an internally annotated pediatric sleep corpus. Our experiments demonstrate that fine-tuned discriminative models consistently provide higher classification accuracy, lower inference latency, and more robust span-level identification than either prompted or fine-tuned generative LLMs, given adequate training data. We conclude by suggesting future research directions: refining span extraction methods, mitigating biases in model performance, and exploring advanced chain-of-thought prompting techniques.

Introduction

Sleep health is a complex, multidimensional construct involving timing, duration, efficiency, and behavioral patterns. While vital for patient care and epidemiological research, this information is rarely stored in discrete database fields. Instead, it is buried in free-text clinical notes. Traditional Natural Language Processing (NLP) relied on rigid, rule-based systems or simple keyword matching, which struggle with the linguistic variety and nuance of clinical narratives. This study investigates whether modern LLMs—either through dynamic prompting or task-specific fine-tuning—can reliably automate the "phenotyping" of sleep health at scale.

Table 2: The MSH classes annotated in the NCH-Sleep dataset, including descriptions and prevalence.

Methods

The researchers systematically compared three primary approaches for Boolean classification across nine distinct sleep dimensions:

  1. Dynamic Few-Shot Prompting: Utilizing generative models (Llama-3.1-8B and BioMistral-7B) without specific training. An external encoder was used to retrieve the most relevant training examples for each specific note to include in the prompt.
  2. Fine-tuned Generative LLMs: Training Llama-3.1-8B on task-specific data using efficient techniques like LoRA and 4-bit quantization.
  3. Fine-tuned Discriminative Classifiers: Training ModernBERT-base (an encoder-only model) to directly predict the binary presence of sleep health dimensions.

Datasets

The study utilized three primary data sources to evaluate model performance across different clinical contexts. Publicly available de-identified clinical notes from MIMIC-III (MDACE and SDOH) served as "out-of-task" proxy data, while the NCH-Sleep dataset provided high-quality, expert-annotated pediatric clinical notes.

Table 1: Basic metrics of datasets used, including note length and class distribution.

Key Results and Insights

Performance Benchmarks

  • Top Performer: The fine-tuned ModernBERT (trained on a "Mix" of MIMIC and NCH data) achieved the highest F1 score (0.4268) and AUC (0.7083). Smaller, encoder-only models proved more effective at "fitting" the specific nuances of clinical sleep mentions than massive generative models.
  • Latency Gap: ModernBERT demonstrated clear superiority for clinical deployment, processing notes in 0.076 seconds—roughly 250x faster than prompted Llama-3.1-8B (18.858s).
  • Prompting Utility: In zero-labeled-data scenarios, prompted LLMs achieved a moderate F1 of ~0.35, providing a useful baseline for initial data exploration despite not being production-ready.

Challenges in Classification

Certain constructs, specifically "Sleep Efficiency" and "Daytime Sleepiness," proved the most difficult to classify. These dimensions often involve longer, more complex textual representations compared to simpler mentions like "Sleep Medication." Furthermore, fine-tuned generative LLMs occasionally suffered from "hallucinations," failing to output valid Boolean answers—a problem the discriminative classifier avoids by design.

Linguistic and Demographic Bias

Performance varied significantly by patient demographics. Models performed better on notes for female and White patients, suggesting that clinical documentation for these groups may be more explicit or consistent, while notes for male and minority patients might contain more ambiguity or linguistic heterogeneity.

Conclusion

While massive generative models are versatile "cold-start" solutions, smaller, fine-tuned discriminative classifiers like ModernBERT offer superior accuracy and significantly lower latency for real-world clinical deployment. Future work should focus on exploring Chain-of-Thought (CoT) prompting to improve reasoning for subtle constructs and testing newer architectures like State-Space Models (SSMs) to further optimize inference speed.

Summary updated

Abstract

Abstract Accurate multidimensional sleep health (MSH) information is often fragmented and inconsistently represented within hospital infrastructures, leaving crucial details buried in unstructured clinical notes rather than discrete fields. This inconsistency complicates large-scale phenotyping, secondary analyses, and clinical decision support regarding sleep-related outcomes. In this work, we systematically explore contemporary natural language processing techniques, prompt-based large language models (LLMs) and fine-tuned discriminative classifiers, to bridge this critical gap. We evaluate performance on extracting nine key MSH dimensions (timing, duration, efficiency, sleep disorders, daytime sleepiness, interventions, medication, behavior, and satisfaction) from clinical narratives using public datasets (MIMIC-III derivatives) and an internally annotated pediatric sleep corpus. Initially, we assess generative LLM performance using dynamic few-shot prompting, analyzing impacts from varying prompt structures, example quantity, and domain-specificity without explicit task-specific fine-tuning. Subsequently, we fine-tune generative LLM architectures on both in-task and out-of-task data to quantify performance improvements and limitations. Lastly, we benchmark these generative approaches against encoder-based discriminative classifiers (ModernBERT), designed to directly estimate binary presence of each MSH class within full clinical notes. Our experiments demonstrate that fine-tuned discriminative models consistently provide higher classification accuracy, lower inference latency, and more robust span-level identification than either prompted or fine-tuned generative LLMs, given adequate training data. Nonetheless, generative LLMs retain moderate utility in low-data scenarios. Importantly, our results highlight persistent challenges, including difficulty extracting subtle sleep constructs such as sleep efficiency and daytime sleepiness, and biases associated with patient demographics and clinical departments. We conclude by suggesting future research directions: refining span extraction methods, mitigating biases in model performance, and exploring advanced chain-of-thought prompting techniques to achieve reliable, scalable MSH phenotyping within real-world clinical systems.
BibTeX
@misc{hussain2025empirical,
  author = {Syed-Amad Hussain and Ariana Calloway and Joseph Sirrianni and Eric Fosler‐Lussier and Mattina Davenport},
  title = {{Empirical Review of LLM-driven Classification of Multidimensional Sleep Health Mentions from Free-Text Clinical Notes}},
  year = {2025},
  doi = {10.1101/2025.06.04.25328983},
  howpublished = {medRxiv},
}
Figure