HALO: Hierarchical Evidence Selection and Reinforcement Learning for Allocating Budgeted Label Optimization from Longitudinal Clinical Records

Huaiyuan Ying, Feiqing Huang, Ryumei Nakada, Katherine P. Liao, Tianxi CaiarXiv 2026 · arXiv

TL;DR

HALO is a reinforcement learning framework designed to solve a critical bottleneck in clinical machine learning: how to efficiently label massive, messy longitudinal health records under a tight budget. By jointly optimizing what evidence to read and which patients deserve expensive labels, HALO outperforms standard active learning methods, achieving superior classification performance on external datasets while spending less.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a treasure trove for medical research, but they are notoriously difficult to work with. A single patient’s history might span years of notes, lab results, and fragmented documentation. Processing all this text with top-tier Large Language Models (LLMs) is prohibitively expensive, and hiring clinicians to manually label every case is even worse.

Existing "active learning" techniques usually focus on the second half of the problem: deciding which patients are the most "informative" to label. However, they typically assume the data for those patients is already cleaned and summarized. In reality, the selection of evidence from a long medical record and the allocation of a labeling budget are two sides of the same coin. If you don't select the right evidence, even an expert labeler might give you the wrong answer.

Methods

The authors introduce HALO (Hierarchical Evidence Selection and Reinforcement Learning for Allocating Budgeted Label Optimization). The pipeline operates in two main stages, optimized jointly via Proximal Policy Optimization (PPO):

  1. Hierarchical Evidence Selection: This is a coarse-to-fine filtering process.
  2. Note Level: A frozen scorer (KOMAP) filters out irrelevant clinical notes based on phenotype-specific scores.
  3. Segment Level: A trainable actor identifies specific, informative segments within those notes using clinical entity embeddings and attention mechanisms.
  4. Summarization: The selected segments are condensed into a compact patient summary.
  5. Budgeted Label Allocation: A "patient-worthiness" scorer (based on a T5 encoder) analyzes the compact summary to decide which label source to use for that specific patient:
  6. Tier 0 (Free): Weakly supervised labels (e.g., from simple keyword rules).
  7. Tier 1 (Silver): Labels from a mid-tier LLM (e.g., Llama 3 8B).
  8. Tier 2 (Gold): Labels from a high-reasoning LLM (e.g., GPT-5) or expert review.

The system learns to reserve the "Gold" budget for complex, ambiguous cases where high-fidelity labels provide the most value for training a downstream classifier.

Overview of the HALO architecture, showing the flow from raw longitudinal notes to hierarchical selection, and finally to budget-aware label allocation.

Datasets

  • Training: The model was trained using MIMIC-IV, a large, de-identified dataset of over 10,000 patients with heart-failure-related records.
  • Acquisition Evaluation: The policy was tested on the MedAlign cohort (106 patients) to measure how well the acquired labels matched a high-quality proxy reference.
  • Downstream Evaluation: To test real-world utility, a classifier trained on HALO-acquired labels was evaluated on an external Mass General Brigham (MGB) Biobank cohort (299 patients) against clinician-adjudicated labels for HFpEF (Heart Failure with preserved Ejection Fraction).

Key Insights and Ideas

  • Coupled Decisions: The core "aha" moment is that evidence selection and label acquisition must be optimized together. By filtering out noise early, the label-allocation policy can more accurately judge a patient's "worthiness" for an expensive label.
  • Asymmetric Noise Theory: The paper provides a theoretical framework showing that false-positive and false-negative errors in acquired labels shift downstream model performance in different ways. HALO is designed to minimize the "feature-weighted label corruption" that hurts model generalization.
  • Focus on the Boundary: The RL policy naturally learns to spend the budget on "hard boundary" cases—patients whose records are complex enough that a cheap label would likely be wrong, but a gold label would provide a clear training signal.

Key Results

In a case study focused on identifying HFpEF:

  • Label Accuracy: HALO achieved the highest F1 score for acquired labels on the MedAlign dataset in six out of eight tested budget levels.
  • Downstream Performance: When a classifier was trained on these labels and tested on the external MGB cohort, HALO dominated at lower budgets (15–60 cost units).
  • Efficiency: At higher budgets, HALO remained highly competitive, staying within 0.01 F1 of the best-performing (and often more computationally expensive) comparators.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Soft Budget Constraints: The current reward function uses a penalty for overspending rather than a hard cap, leading to occasional budget overruns of 3–5%.
  • Fixed Taxonomy: The model currently assumes a fixed set of three label sources (Free, Silver, Gold). Future versions could adapt to a continuous spectrum of costs and fidelities.
  • Domain Adaptation: The system relies on some pretrained components (like KOMAP) which might introduce biases when moving to entirely different medical specialties or hospital systems.

Summary updated

Abstract

Learning from extensive longitudinal clinical records--spanning repeated encounters, evolving conditions, and heterogeneous documentation--under a finite labeling budget requires solving two coupled problems: deciding what evidence to keep and deciding which patients deserve higher-fidelity labels. Prior work on active learning and multi-fidelity label acquisition mainly addresses the second problem, while treating the evidence presented to the acquisition policy as fixed. We present HALO, a Hierarchical Evidence Selection and Reinforcement Learning pipeline for Allocating Budgeted Label Optimization from Longitudinal Clinical Records. This framework jointly learns what to read and what label source to acquire under budget. HALO first selects informative evidence from longitudinal records and then uses a budget-aware policy to allocate free-, silver-, and gold-fidelity label sources. For frozen policies, our theory characterizes asymmetric acquired-label channels and gives a budget-constrained finite-sample gold-risk bound for a common downstream logistic model. In a heart-failure case study, HALO attained the highest MedAlign acquired-label F1 at six of eight target budgets. A downstream classifier trained on HALO-acquired labels attained the highest external MGB F1 at budgets 15-60 and remained within 0.01 of the best comparator at budgets 75-120.
BibTeX
@misc{ying2026halo,
  author = {Huaiyuan Ying and Feiqing Huang and Ryumei Nakada and Katherine P. Liao and Tianxi Cai},
  title = {{HALO: Hierarchical Evidence Selection and Reinforcement Learning for Allocating Budgeted Label Optimization from Longitudinal Clinical Records}},
  year = {2026},
  eprint = {2611.00986},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00986},
}
Figure