Not Discussed Is Not Negative: Topic Coverage as a Missingness Signal in Clinical Text

Studies that mine electronic health records (EHRs) after the fact usually read a clinical note as a bag of findings. If a symptom is not written down, the patient is coded as not having it. This fails for the sickest patients, who often get the thinnest notes. When Bias Associated with Mining Electronic Health Records automatically rebuilt the Fine pneumonia severity index from hospital records, the lowest-risk class (Class I) had nearly double the mortality of Class II. Patients who died quickly in the emergency department had almost empty records, and the software read those empty records as a clean bill of health. We propose to keep three states apart for every clinical topic: the topic was discussed and the finding was present, the topic was discussed and the finding was absent, or the topic was never discussed. The third state is detected with topic segmentation, meaning a model that splits a conversation or note into contiguous episodes, each about one subject. Instead of being filled with a default, the "never discussed" state enters the downstream risk or causal model as its own variable.

The opportunity lies where the two literatures fail to meet. Dialogue segmentation research (SuperDialseg, DialSTART, SumSeg) partitions text that already exists and has never asked what the missing episodes mean. Work on retrospective-study bias documents that whether something gets recorded depends on how sick the patient is (colitis cohort, gastric cancer imputation). That work has no tool for finding, inside unstructured text, which parts of a standard evaluation never happened. Our claim is that an episode-level coverage map turns documentation sparsity from a silent source of error into a measured variable.

Core Claim

For each patient encounter we build a coverage vector over a fixed inventory of expected clinical topics. Examples include history of present illness, review of systems, vital signs, respiratory exam, mental status, medication review, and goals-of-care. Each entry is one of three values: present, negative, or not covered. Coverage is decided at the episode level, not by keyword hits. A topic counts as covered only if a segment is substantially devoted to it. A passing templated phrase inside some other segment does not count.

The hypothesis has two parts. First, adding these indicators to a severity or confounder model will repair the documentation paradox. Concretely, observed mortality will rise monotonically across the reconstructed severity classes, and the bias in treatment-effect estimates will shrink when recording depends on severity. Second, the episode-level definition will beat cheaper proxies for documentation density. If it does not beat them, we will say so and conclude that the cheap proxies are enough.

This is not the closest prior work recombined. CLUSTER2SENT groups doctor–patient utterances into clusters in order to write SOAP-note sentences. It produces text, never asks what is absent, and never feeds a causal estimate. Bayesian Topic Regression learns text topics jointly with a regression so that the effect estimate is not biased by fitting the topics separately. It still treats each document as an exchangeable bag of words, so a topic that is absent looks the same as a topic that is weak. SADD uses segmentation to "debias," but the bias it targets is the order in which a sequence decoder emits its outputs, not observational confounding.

Research Program

The work has three components.

The first is a clinical segmenter that does not depend on section headers. Several sources motivate the design. Dialogue structure follows the structure of the underlying task (user–adviser dialogue study). Topic-concentrated fragments make more stable units than whole histories (topic-aware dialogue comprehension). Shifts are often signaled by discourse cues rather than repeated words (Museli).

For the method itself, we reuse the SuperDialseg recipe of turning an external structure into silver boundary labels. SuperDialseg used transitions between source-document sections as boundary proxies. In clinical text, the proxies come from two places. For doctor–patient transcripts, the Abridge-style utterance-to-SOAP-sentence links reported for CLUSTER2SENT give us which utterances belong to which note section. For EHR notes, we use the section headers clinicians already type, and then strip them so the model must learn boundaries from content alone.

The segmenter is a RoBERTa boundary classifier fine-tuned on these proxies, with DialSTART-style contrastive pretraining for the unlabeled remainder. After segmentation, a lightweight classifier maps each segment to a topic in the inventory and labels its polarity. One design choice comes directly from the bias literature. SumSeg smooths its similarity curve with a Savitzky–Golay filter to suppress short blips. In clinical text a one-turn episode, such as a single exchange about an allergic reaction, may be the most informative event in the encounter. We therefore keep short segments, and we ablate the filter.

The second component is the missingness-aware downstream model. Coverage indicators and polarities enter the model together with structured covariates. We use a logistic outcome model for severity calibration, and for causal questions a propensity model in the large-scale, many-covariate style of LSPS. Because the indicators are deterministic functions of text, and the segmenter is never fitted on the outcome, we avoid the two-step leakage that BTR proves biases effect estimates when latent topics are fitted separately.

A related argument applies to propensity models. LSPS adjusts for an unmeasured confounder only when that confounder can be recovered from the observed covariates, a property the LSPS paper calls "pinpointability." We argue that coverage indicators add exactly that kind of recoverability for one specific hidden variable: the acuity that caused the evaluation to be cut short. The EHR bias scoping review finds referral and admission biases to be the most frequent category (35% of the 116 studies it covers). These workflow-driven biases are what coverage is meant to capture.

The third component is an evaluation design with known ground truth. Real retrospective data never reveal the true confounder, so we build a semi-synthetic setting. We start from densely documented encounters, where all topics are covered and true severity can be read off reliably. We then delete whole episodes with a probability that rises with severity. This reproduces the missing-not-at-random pattern described in the colitis and pancreatic-surgery studies (PSM analysis). Treatment and outcome are simulated from true severity, so the true effect is known.

Baselines

The naive reference point is the flat extraction pipeline Hripcsak et al. audited. Findings are extracted from the whole note, and absent findings default to negative. On real data we expect to reproduce the inverted Class I/II mortality. The file gives only the qualitative "nearly double" result and no exact rate, so the target we must beat is a qualitative inversion rather than a number.

The strongest learned competitor is a monolithic clinical encoder: PubMedBERT reading the whole unsegmented note, as in the MIMIC-IV clinical-outcome benchmark (COP-IV). That paper reports AUROC 86.86% and PR-AUC 17.24% for emergency-department diagnosis prediction, against roughly 14–15% PR-AUC for the other baselines. We will run PubMedBERT on the same COP-IV splits, with the same fine-tuning budget and the same structured covariates. It counts as beaten only if adding coverage indicators improves calibration within the lowest-predicted-risk stratum and makes mortality rise monotonically across classes, while not lowering PR-AUC below 17.24%.

The strongest causal competitor is LSPS on structured codes alone. It gets the same patients, the same regularization search, and the same empirical calibration with negative controls from the OHDSI benchmark. That benchmark reports that calibration raised confidence-interval coverage for a new-user cohort design in the CCAE database from 0.73 to 0.89. In the semi-synthetic setting, LSPS is beaten if adding coverage indicators reduces absolute bias in the estimated effect and brings 95% interval coverage closer to nominal.

For segmentation itself, the reference is the SuperDialseg RoBERTa. It reports F1 0.784 in-domain and 0.660 when transferred to DialSeg711. DialSTART reports Pk 17.86% on DialSeg711, where Pk is the probability that two nearby turns are wrongly judged to be in the same or different segments. Our clinical segmenter should approach these figures on held-out clinical silver boundaries. If it falls far below them, the coverage map is unreliable no matter how well the downstream model performs.

Three ablations isolate the key ingredient. The first replaces episode-level coverage with mention-level coverage: a topic counts as covered if any keyword or concept appears anywhere. The second replaces coverage with scalar documentation density, using note length and segment count. The third restores Savitzky–Golay smoothing.

Why not just…?

Why not just use note length or note count as a documentation-density covariate? This is the most serious threat. The documentation paradox is at heart a story about short notes, so length may capture most of the signal. Our proposal stands only if the direction of missingness matters, not just its amount. For example, a long note with no respiratory exam should mean something different from a long note with no goals-of-care discussion. The density ablation settles the question directly. If length matches coverage on calibration and on effect bias, we will report that the simple covariate is sufficient and drop the segmentation machinery.

Why not just parse section headers with rules? On templated EHR notes, header parsing is a strong and cheap way to measure coverage. We treat it as an upper bound on those notes, not as a competitor. The case for a learned segmenter rests on text without reliable headers: doctor–patient transcripts, free-text emergency-department notes, and templated headers left in place over empty content. We report results separately for header-bearing and headerless text. If the learned segmenter adds nothing on header-bearing notes, that is expected and not a failure.

Why not just prompt a long-context LLM with "was topic X evaluated?" SuperDialseg found zero-shot ChatGPT segmentation well below the supervised RoBERTa. CLUSTER2SENT and SADD both report that unconstrained generation on long dialogues drifts from the source. Still, per-topic yes/no questions are easier than placing boundaries, so we include a zero-shot LLM coverage extractor as an extra baseline within the same evaluation budget. Its main expected weakness is that it gives no attributable span to audit.

Why not just impute the missing findings, or keep only complete records? Multiple imputation assumes that whether a value is recorded does not depend on the unrecorded value itself, given the observed data. This assumption is exactly what the gastric-cancer and colitis cohorts call into question. Restricting to complete records was tested directly by Hripcsak et al.: it suppressed mortality in Class V, the sickest group, because those patients died before a discharge summary was written. Both appear in the semi-synthetic experiment, where their bias can be measured against the known truth.

Proposed Experiments and Decisive Checks

The first experiment is the semi-synthetic missing-not-at-random study, run on densely documented MIMIC-IV notes and on Abridge-style transcripts if access permits. We vary how steeply deletion probability rises with severity, and we measure effect bias and interval coverage for each method. The decisive pattern is that the flat, imputation, and complete-case methods degrade as the dependence on severity grows, while coverage-augmented models stay flat. Episode-level coverage must also beat mention-level coverage and length.

The second experiment reconstructs a pneumonia severity score on MIMIC-IV emergency-department and ICU records. We ask whether the Class I/II mortality inversion appears with flat extraction and disappears once coverage indicators let "not assessed" patients be modeled separately. A related test checks whether such patients carry mortality closer to the higher classes than to Class I.

The third experiment reports intrinsic segmentation quality, measured by Pk, WindowDiff, and boundary F1, on clinical silver boundaries with headers stripped. To check that the segmenter transfers, we also report results on DialSeg711 and SuperDialseg.

The proposal fails cleanly in two ways. If scalar length matches coverage in the first two experiments, the conclusion is that segmentation adds nothing here. If segmentation quality on clinical text falls well below the 0.660 out-of-domain F1 that SuperDialseg reports, the coverage map is too noisy to trust.