Abstract: Accurate detection of multidimensional sleep health (MSH) information from electronic health records (EHRs) is critical for improving clinical decision-making but remains challenging due to sparse documentation and class imbalance. This study investigates whether integrating expert-guided annotations and keyword-based heuristics with large language models (LLMs) enhances the extraction of nuanced MSH indicators from clinical narratives. Using a novel, expertly annotated dataset (NCH-Sleep), we trained and evaluated models to classify clinical notes across nine clinically relevant MSH categories. Our baseline model demonstrated substantial predictive capability using raw text alone. Incorporating manually annotated spans (oracle annotations) dramatically improved performance, highlighting the benefit of targeted expert guidance. Additionally, employing curated keyword annotations within varying context windows significantly enhanced model interpretability while retaining strong predictive accuracy. Through detailed bias analyses, we identified consistent performance across demographics and clinical settings, although specific disparities underscored the importance of balanced expert oversight. Our findings emphasize the value of expert-informed supervision and heuristic approaches in building scalable, interpretable clinical NLP systems for sleep health classification.
Introduction
Extracting multidimensional sleep health (MSH) data from Electronic Health Records (EHRs) is a critical yet challenging task. Clinical narratives are often unstructured, jargon-heavy, and plagued by "class imbalance"—where specific information appears in only a small fraction of notes. While traditional keyword searches lack nuance and large language models (LLMs) often lack interpretability, this study explores how integrating expert-guided "signals"—such as curated keyword windows and manually highlighted text spans—can bridge the gap between scalability and clinical trust.
Methodology
The researchers approached MSH extraction as a boolean classification task, treating each of the nine MSH categories as an independent "True/False" query. They evaluated two primary architectures: ModernBERT-large (395M parameters, encoder-only) and Gemma-2-2b (2B parameters, decoder-only).
Three experimental configurations were tested to assess the impact of expert insight:
- Baseline: Models received raw clinical notes and brief class descriptions.
- Oracle Guidance: Models were provided with "oracle spans"—specific text segments identified by experts as diagnostic evidence.
- Keyword Heuristics: Models were guided by a list of 348 sleep-related keywords, using varying context windows (e.g., 40 to 640 characters) to balance interpretability and performance.
Dataset Description
The study introduced the NCH-Sleep dataset, comprising 1,200 clinical documents from Nationwide Children’s Hospital (2018–2023). These notes span primary care, behavioral health, healthy weight, and school-based programs. Each note was expertly annotated for nine MSH categories, with a test set of 300 notes undergoing rigorous double-expert adjudication.
Results and Discussion
The decoder-only Gemma-2-2b model consistently outperformed the encoder-based ModernBERT-large, achieving a baseline F1 score of 0.62 and an AUROC of 0.86. This suggests that the larger scale and pre-training of decoder models provide a more robust "world model" for understanding clinical nuances.
A key finding was the superiority of binary classification over multi-label approaches. Training separate models for each class allowed for the inclusion of specific class descriptions in prompts, which acted as a crucial guide for the models.
Performance significantly improved with expert guidance. When using "Oracle Only" text, the F1 score rose to 0.82 (AUROC 0.97). Notably, the keyword-based heuristic with a 640-character window (keyword_only_640) achieved an F1 of 0.75, effectively narrowing the gap between simple keywords and resource-intensive human annotations. Bias analysis also revealed that restricting models to expert-identified oracle spans helped mitigate performance disparities across demographic groups, underscoring the value of expert supervision in ensuring algorithmic fairness.
Conclusion
This research demonstrates that LLMs can effectively classify complex clinical data when guided by expert-derived heuristics. By focusing on high-yield text segments, clinical NLP systems can achieve high accuracy while remaining auditable and interpretable for practitioners. Future work will explore dynamic keyword expansion and automated span prediction to further scale these insights across diverse hospital systems.
Summary updated