In most machine learning domains, data labeling is a commodity. However, in healthcare, the "expertise wall" makes manual annotation a primary bottleneck. Labeling a single clinical note for social determinants of health (SDoH) or genetic variants requires hours of attention from highly trained clinicians, making it prohibitively expensive to scale supervised models across diverse institutions.
To bypass this bottleneck, researchers are shifting away from brute-force manual labeling toward strategies that maximize the utility of expert time or replace it entirely with automated proxies.
Strategic Sampling via Active Learning
The most direct way to lower the annotation burden is to ensure that experts only label the most informative data points. This is the core intuition behind active learning: rather than labeling a random subset of data, the system identifies samples where the model is most uncertain or where the data is most likely to represent rare, critical classes.
Jenna Wiens et al. demonstrated this in Active Learning Applied to Patient-Adaptive Heartbeat Classification. Because every patient’s heart rhythm is unique, a general model often fails. By strategically asking cardiologists to label only the most "informative" heartbeats for a specific patient, the authors achieved state-of-the-art accuracy with 90% less manual effort. Similarly, Mari Ostendorf et al. used active learning in Annotating Social Determinants of Health Using Active Learning, and Characterizing Determinants Using Neural Event Extraction to bias data collection toward rare risk factors like homelessness, ensuring that the resulting models could handle nuanced, low-frequency events without requiring a massive, balanced dataset.
LLMs as Expert Force Multipliers
Recent advances in Large Language Models (LLMs) have introduced a new paradigm: using general-purpose models to perform the "first pass" of annotation. This shifts the human expert's role from creator to editor, which is significantly faster and less fatiguing.
The authors of LLMs Accelerate Annotation for Medical Information Extraction found that using LLMs to generate base annotations reduced human labor by 58% while maintaining high ground-truth quality. This works because LLMs, despite not being specifically trained on a hospital's local data, possess enough general medical knowledge to handle standard terminology, leaving only the most complex nuances for human review.
In some cases, the LLM can even act as the final annotator. Seiji Shimizu et al. proposed SALA in Exploring LLM Annotation for Adaptation of Clinical Information Extraction Models under Data-sharing Restrictions, where an LLM corrects the mistakes of a smaller, specialized model. This human-free adaptation process allows hospitals to fine-tune models to their local linguistic quirks without violating data-sharing restrictions or hiring expensive annotators.
Synthetic Data and Silver Standards
When real data is too sensitive to share or too expensive to label, researchers generate "silver standard" datasets—large-scale corpora labeled by automated systems rather than humans.
Ian Lewin et al. illustrated this in Deriving an English Biomedical Silver Standard Corpus for CLEF-ER by harmonizing the outputs of six different automated systems. By using a "centroid" voting method, they created millions of high-quality labels for entities in patents and drug labels, bypassing the need for human experts entirely.
Taking this a step further, LLMs can be used to generate entirely synthetic clinical narratives. Zonghai Yao et al. developed the SynthEHR-Eviction pipeline, described in SynthEHR-eviction: enhancing eviction SDoH detection with LLM-augmented synthetic EHR data, which generates and labels synthetic notes. Models trained on this synthetic data achieved an 89% F1-score, outperforming even GPT-4o in detecting complex social factors. This works because the synthetic generation process can be controlled to produce "hard examples" that force the model to learn subtle distinctions that might be rare in a naturally occurring dataset.
Bypassing Labels with Deep Representations
Finally, some researchers argue that the bottleneck can be avoided by moving away from discrete labels altogether. In Leveraging Deep Representations of Radiology Reports in Survival Analysis for Predicting Heart Failure Patient Mortality, Hyun Gi Lee et al. bypassed traditional rule-based clinical labels. Instead of training a model to predict a human-defined category, they used raw BERT-based embeddings of radiology reports to predict patient mortality directly.
This approach suggests that the "expert bottleneck" is partly a byproduct of our desire to force complex clinical narratives into simple, human-readable tags. By allowing models to learn directly from the high-dimensional representations of the text, we may capture prognostic signals that human experts—and their labels—might otherwise overlook.
Key Foundational Work
Extracting Diagnoses and Investigation Results from Unstructured Text in Electronic Health Records by Semi-Supervised Machine Learning (2012) — contributed 1 concept elements to this direction.
Active Learning Applied to Patient-Adaptive Heartbeat Classification (2010) — contributed 1 concept elements to this direction.