TL;DR
OmitEHR is a reference-free framework that detects clinically significant information missing from LLM-generated hospital summaries. By training a contextual model on decades of historical clinician-written notes, it learns to identify "salient" facts and flags omissions 6x cheaper and nearly 2x faster than using LLM judges alone.
What Problem Does This Paper Address?
As health systems deploy Large Language Models (LLMs) to summarize Electronic Health Records (EHRs), a critical failure mode has emerged: omissions. While researchers often focus on "hallucinations" (making things up), clinicians find that LLMs are far more likely to simply leave out crucial details—like a withheld medication or a specific diagnostic finding—which can lead to medical errors.
The challenge in detecting these omissions is determining salience. A patient's record might contain thousands of sentences; a detector must decide which ones are important enough to belong in a summary without having a "gold standard" reference summary available at the time of care. Existing methods either use expensive LLMs to re-read the entire chart for every summary (which doesn't scale) or use simple heuristics that fail to capture clinical nuance.
Methods
OmitEHR decouples omission detection into two distinct steps: Salience Prediction and Coverage Assessment.
- Learning Salience from History: The authors leverage the fact that clinicians have been writing "Brief Hospital Course" (BHC) summaries for decades. They used an LLM to align 2.9 million historical source-note sentences with their corresponding clinician-written summaries to create "silver-standard" labels (Directly Included, Indirectly Related, or Not Related).
- The Model: They trained
OmitEHR-contextual, a bidirectional Transformer built on top of BioClinRoBERTa. The model incorporates both semantic embeddings and structural features (e.g., note type, position in the record, time since admission). Crucially, it considers the context of the entire admission to decide if a sentence is salient. - The Pipeline:
- At inference, the trained model scores every sentence in a new patient's record for salience.
- Only the Top-K (e.g., 100) highest-scoring sentences are sent to an LLM "judge."
- The LLM judge performs a simple coverage check: "Is the information in this specific salient sentence reflected in the generated summary?"
- Uncovered salient sentences are flagged as candidate omissions.
Datasets
- Primary Dataset: 5,000 internal medicine admissions from a large academic medical center.
- Scale: Approximately 2.9 million source sentences.
- Validation Set: A subset of 300 sentences was independently annotated by two board-certified internal medicine clinicians to validate the LLM-generated "silver" labels.
- Evaluation Set: 25 admissions were used for a blinded clinician review of the final omission candidates.
Key Insights and Ideas
- Salience is Grounded in Action: Instead of asking an LLM "what is important?" via a prompt, OmitEHR learns from the actual behavior of physicians over years of practice. This grounds the model in real-world clinical judgment.
- Computational Efficiency via Decoupling: By using a compact, local Transformer for salience and reserving the expensive LLM only for coverage checks on a fixed number of candidates (Top-K), the cost of the system does not grow linearly with the length of the medical record.
- Context Matters: The authors found that "contextual" models significantly outperformed "frozen" or "fine-tuned" sentence-level models. A sentence's importance often depends on whether it was repeated later or rendered irrelevant by a subsequent clinical event.
Key Results
- Superior Salience Detection:
OmitEHR-contextualachieved an AUROC of 0.828, outperforming a zero-shot GPT-4o-mini judge (0.806) and various lexical/heuristic baselines. - Clinician Validation: In a blinded head-to-head review, OmitEHR surfaced a mean of 5.64 clinically meaningful omissions per summary, compared to only 3.32 for a pipeline using an LLM to judge salience.
- Speed and Cost: The pipeline is 5.6x to 6.7x cheaper ($0.03–$0.13 per summary) and 1.6x to 1.7x faster than the LLM-judge baseline.
- Reliability: OmitEHR surfaced at least one omission confirmed by both reviewing clinicians in 24 out of 25 test admissions.
Limitations and Future Work
- Generalization: The study was conducted at a single academic center within internal medicine. Future work is needed to see how well these salience patterns transfer to other specialties or hospitals.
- Recall Measurement: The study measures precision (how many flagged omissions were real), but measuring recall (how many real omissions were missed) remains difficult because it requires clinicians to manually audit thousands of source sentences per patient.
- The "Silver Label" Gap: The model is trained on LLM-aligned historical data. While agreement with clinicians was moderate-to-high, the labels are not perfect.
- Closing the Loop: The authors suggest that detected omissions could be automatically fed back into the LLM summarizer to trigger a "revision" step, creating a self-correcting summarization loop.
Summary updated