Hospital prediction models increasingly combine chest X-rays, structured records, and clinical notes. The rarest and most dangerous conditions, however, are exactly where these models fail. One possible reason is that the text encoders used in such systems average a note into a single dense vector. That averaging blurs the few exact tokens that often signal a rare condition: a specific drug name, an abbreviation, a lab term. The signal is further buried because gradients from common labels dominate training. Older sparse text representations keep those tokens intact. The best known is TF-IDF, which scores each word by how often it appears in a document, discounted by how common it is across all documents. Sparse representations are poor at context, negation, and synonymy, but they preserve exactly the surface detail that dense models smooth away.
This note proposes a specific, falsifiable claim. In multimodal clinical prediction, the value of a sparse lexical channel is concentrated in low-prevalence labels, and it grows as prevalence falls. The channel should weight words by how strongly they indicate each label, rather than by corpus rarity alone. We test this on the MIMIC-IV/MIMIC-CXR fusion benchmark, adding a per-label sparse text score to strong image-plus-record models. We measure gains label by label with metrics that do not hide minority-class failure. If a well-tuned dense model with an imbalance-aware loss closes the gap, the claim is refuted. That outcome would be a useful negative result for the field.
Core Claim
Two findings motivate the claim. First, When BERT Fails - The Limits of EHR Classification showed that ClinicalBERT embeddings of accumulating MIMIC-III notes grew more "pessimistic" over a stay. Misclassified survivors rose from 16,206 to 16,908 while correct deaths barely moved, and 25% of survivors formed a high-confidence false-death cluster marked by heavy documentation and frequent readmissions. The dense encoder was reading documentation volume, not acuity. Second, When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion (CareBench) found three things. Fusion gains are small, with best-case phenotyping AUPRC rising from 0.472 to 0.495. The gains are concentrated in a few conditions. And naive fusion collapses when most images are missing. Neither paper uses free text alongside images. Neither asks whether a lexical channel helps specifically where labels are rare.
The evidence that sparse and dense text are complementary comes from outside medicine. HTMS@DravidianLangTech 2025 concatenated compressed TF-IDF vectors with BERT embeddings for abusive-comment detection, raising macro F1 from 0.59 to 0.62. WUY at SemEval-2020 Task 7 combined BERT with a Naive Bayes–SVM (NB-SVM), a linear classifier whose word weights are the log-ratio of a word's frequency in positive versus negative examples. That combination reduced humor-score RMSE to 0.52429, against 0.56439 for NB-SVM alone. Both papers report averaged gains on short social-media or headline text. Neither says anything about where across the label distribution the gain lives, and neither involves images or clinical records.
The claim here is narrower and testable. The benefit of the lexical channel, measured as a per-label AUPRC difference against the same model without it, should be negatively correlated with label prevalence. Class-conditional word weights should beat class-blind IDF weights, and the gap between them should also be largest on rare labels.
The reasoning behind the second part is simple. IDF upweights anything rare, including typos and irrelevant proper nouns. A per-label log-count ratio upweights only tokens that separate that label's positives from its negatives. Under heavy imbalance, those tokens are nearly invisible to a dense encoder trained with a shared loss.
Research Program
The model keeps a standard multimodal backbone: an image encoder for the chest X-ray, an encoder for the structured record, and a dense note encoder, fused as in the strongest CareBench configurations. To this backbone we add, for each label $k$, a sparse lexical score $s_k(d)$. This score is a linear function of the note bag $d$, built from word and bigram counts with sublinear term frequency and length normalization. Its weights are $r_k$, the smoothed log-ratio of token frequencies in label-$k$ positives versus negatives, as in NB-SVM.
The final logit for label $k$ is
$$z_k = z_k^{\text{dense}} + \alpha_k \, s_k(d),$$
where $z_k^{\text{dense}}$ is the backbone's logit and $\alpha_k \ge 0$ is a learned per-label weight. The weight $\alpha_k$ is shrunk toward a shared value, so that labels with few positives do not overfit it. The ratios $r_k$ are computed on training folds only.
Two design choices keep the lexical channel from creating new problems. The first is image starvation. CareBench already shows that rich record data starves the image branch, and adding a strong text signal could make this worse. We therefore keep a unimodal auxiliary loss on the image branch. We report the drop in performance when images are removed as a first-class outcome, so that any gain bought by further ignoring the image is visible. The second is missing modalities. The lexical score is additive and needs no image, so we train with random modality dropout. We evaluate on CareBench's base cohort, where 75% of radiographs are missing.
Leakage is the main risk with clinical notes. Phenotype labels in MIMIC are derived from codes that discharge summaries restate. We therefore restrict text to notes timestamped before the prediction cutoff and exclude discharge summaries entirely. We report how many stays keep any usable text under this rule. We treat the radiology report of the current X-ray as a separate, clearly labelled condition, because it restates the image and would muddy the image-versus-text comparison.
The analysis has three parts. The main analysis regresses the per-label AUPRC gain from the lexical channel on log prevalence, across CareBench phenotypes. The second audits which tokens carry the largest $\alpha_k r_k$ weights for rare labels and has a clinician judge whether they are clinically meaningful or artifacts such as note templates or unit names. The third reruns the MIMIC-III mortality cohort from When BERT Fails and asks whether the length-normalized lexical score shrinks the high-documentation false-death cluster. Length-normalized sparse vectors are unaffected by how many notes a stay generates, which dense mean-pooling does not guarantee.
Baselines
The naive reference points are two unimodal models trained on the same stays with the same label splits. One is CareBench's structured-record Transformer, which its authors found beat naive fusion under realistic missingness. The other is a TF-IDF logistic regression on the pre-cutoff notes alone. Any multimodal system that does not beat both on rare-label AUPRC adds nothing.
The strongest existing multimodal competitor is the best CareBench fusion method, including their InfoReg regularizer against modality imbalance, extended with a dense ClinicalBERT note branch. For the text-free version, the reported reference is CareBench's best-case phenotyping AUPRC of 0.495, against 0.472 for unimodal baselines, from When Does Multimodal Learning Help in Healthcare?. No reported result was found on Lacuna for CareBench fusion with clinical notes, so we establish that number ourselves. For the text-plus-record design, How to Leverage Multimodal EHR Data for Better Medical Predictions? supplies the Multimodal Adaptation Gate, which reported acute respiratory failure AUROC 0.792 on MIMIC-III. We reimplement that gate as a second strong fusion baseline.
To make the dense model as strong as possible on imbalance, every dense baseline is also trained with a class-balanced or focal loss and with per-label threshold tuning. The comparison is then with an imbalance-aware dense model, not a straw man. All systems share encoders, data splits, and the number of hyperparameter trials, run on identical compute and with the same five seeds.
Three ablations isolate the key ingredient. One replaces the class-conditional weights $r_k$ with plain IDF weights. One replaces the per-label additive score with HTMS-style concatenation of compressed TF-IDF into the fused vector. One replaces joint training with post-hoc stacking of a separately trained TF-IDF model.
Beating the strongest baseline has a concrete meaning here. The hybrid must show a statistically reliable AUPRC gain, by paired bootstrap across test stays, on the bottom prevalence tercile of labels. Macro AUPRC over the top tercile must not degrade, and the image-removal drop must not shrink.
Why not just…?
Why not just fine-tune ClinicalBERT with a class-balanced or focal loss? This is the most natural alternative, and it is built into every dense baseline rather than argued away. Reweighting changes how much each rare positive counts, but it does not change what a pooled dense representation retains. The volume confound in When BERT Fails arose in balanced 1:1 evaluation, so it is not an imbalance artifact that reweighting would remove. If the focal-loss dense model matches the hybrid in the bottom prevalence tercile, the core claim fails, and we will report it that way.
Why not just stack a TF-IDF classifier on top of the multimodal model, or concatenate TF-IDF features as HTMS did? This is the real threat, because our method is structurally close to stacking. HTMS-style concatenation mixes sparse features into one fused vector. That fused vector is trained under the same majority-dominated loss, so rare-label tokens can again be diluted. Our additive per-label score avoids that dilution. Post-hoc stacking is harder to distinguish from our method, and we run it as an ablation with identical data. If stacking matches joint training, the methodological contribution reduces to class-conditional weighting plus the prevalence-gain relationship. That relationship would still be the finding that matters for how clinical fusion systems should be built.
Why not just use IDF to reweight cross-attention between image regions and note tokens? This is the obvious design at this intersection. It is class-blind: IDF rewards corpus rarity, not relevance to a given diagnosis, and rare typos and template artifacts score as highly as rare findings. We include it as an ablation against class-conditional weights. The prediction is that IDF-weighted variants recover part of the rare-label gain but less than $r_k$, with the difference largest on the rarest labels.
Why not just oversample rare cases or augment text? Red-GAN argues that if enough data exists to train a stable generator, it is often already enough to train the classifier directly. WUY found that easy text augmentation had mixed success at stopping overfitting. Random oversampling of minority stays is nonetheless cheap, so we include it on the dense baseline.
Proposed Experiments
The primary experiment is per-label multi-label phenotyping on CareBench. On the matched cohort of 7,149 stays with complete record and image data, we compare the full hybrid against every baseline and ablation above. On the base cohort of 26,947 stays with 75% of radiographs missing, we repeat the comparison with modality dropout active.
We report per-label AUPRC, the stratified Brier scores $Brier^+$ and $Brier^-$, and the Balanced Brier score recommended by Understanding the impact of class imbalance on the performance of chest x-ray image classifiers. That paper showed AUROC can look near-identical across labels while AUPRC collapses on the rare ones. We also report AUROC for comparability and true-positive-rate gaps across racial groups. CareBench found that fusion can widen those gaps, so a lexical channel that helps rare labels overall but widens them would be a failure.
The decisive check is the slope of the per-label AUPRC gain regressed on log prevalence, estimated across phenotypes with bootstrap intervals. A clearly negative slope, with class-conditional weights beating IDF most on rare labels, supports the claim. A flat slope means the lexical channel is just generic extra capacity. Either way, the image-removal drop tells us whether the gain came from starving the image branch.
The secondary check uses the MIMIC-III subendocardial infarction mortality cohort. Here we count the survivors misclassified as deceased with and without the length-normalized lexical score. We compare against the reported rise from 16,206 to 16,908 false deaths between day 1 and day 8. We also test whether the high-documentation false-death cluster shrinks while correct deaths are preserved. A reduction confined to heavily documented survivors would directly link the sparse channel to the documentation-volume confound. No reduction would mean length normalization does not address it.
References
- When BERT Fails - The Limits of EHR Classification
- When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
- HTMS@DravidianLangTech 2025: Fusing TF-IDF and BERT with Dimensionality Reduction for Abusive Language Detection in Tamil and Malayalam
- WUY at SemEval-2020 Task 7: Combining BERT and Naive Bayes-SVM for Humor Assessment in Edited News Headlines
- How to Leverage Multimodal EHR Data for Better Medical Predictions?
- Red-GAN: Attacking class imbalance via conditioned generation. Yet another medical imaging perspective
- Understanding the impact of class imbalance on the performance of chest x-ray image classifiers