Which Modality Should Learn the Rare Case? Visibility-Gated Focal Loss for Clinical Notes and Vitals

Hospital prediction models increasingly combine two very different inputs: free-text clinical notes and streams of vital signs. The outcomes they try to predict, such as in-hospital death or sepsis, are rare. The standard fix for rare outcomes is focal loss, a training objective that shrinks the penalty on examples the model already gets right so that learning concentrates on the hard ones. In a model with several inputs, focal loss is usually computed once, on the final combined prediction. This note argues that doing so conflates three separate things: how rare a patient's outcome is, how hard the patient is to classify, and whether a given input can see the evidence at all.

When the combined prediction is computed this way, two failures follow. If the notes encoder quickly gets a rare death right, the combined prediction becomes confident and focal loss stops sending gradient to the vitals encoder, even when the vital signs held learnable warning signs. The vitals encoder is starved. The obvious repair is to give each input its own focal loss. That causes the opposite failure: focal loss pushes hardest on the examples an encoder finds hardest, and many of those are deaths that are simply invisible in vitals, for instance a death that follows a comfort-care decision recorded only in the notes. The encoder is then pushed to memorize noise.

We propose to separate the two cases. Each encoder receives its own focal weight, and that weight is multiplied by a gate. The gate comes from an out-of-fold estimate of whether that input alone can predict this patient. Rare patients that are hard but learnable for an input get amplified. Patients that are hard because the input is blind to them get damped.

Core Claim

On imbalanced multimodal clinical prediction, a focal loss computed on the fused prediction causes measurable starvation of the slower modality on minority-class patients. A per-modality focal loss over-fits modality-blind positives. A focal loss gated per modality by cross-fitted learnability fixes both problems, and it does so better than training the modalities separately and stacking them. The claim is falsifiable at each step.

The first falsifiable part is the diagnosis. The vitals encoder's own minority-class precision-recall should be lower under joint fused-focal training than when it is trained alone. The second part concerns the method: the gate should beat both ungated alternatives on AUPRC (area under the precision-recall curve, the metric that tracks rare-positive performance) without degrading calibration.

Relation to Prior Work

Three groups of prior work play different roles here.

The motivating evidence comes from multimodal ICU models trained with plain cross-entropy. Deznabi et al. fuse Bio+Clinical BERT note embeddings with an LSTM over 17 vitals for MIMIC-III mortality. Wang et al. fuse ClinicalBERT with a convolutional time-series model for early sepsis. Multimodal Risk Prediction with Physiological Signals, Medical Images and Clinical Notes compares early, joint, and late fusion on MIMIC-IV-MM. None of the three treats label imbalance at the loss level. Together they establish that notes are the stronger single input for mortality, which makes notes the likely dominant modality.

The general observation that modalities learn at different speeds and that one can starve another comes from Wang et al. and CAT-GS. CAT-GS rebalances gradient budgets across encoders using calibrated unimodal teachers. It does so on audio-visual benchmarks with balanced labels, so it never meets the case where the starved examples are specifically the rare ones.

Prior focal-loss methods supply the ingredients. Lin et al. define focal loss. MARN shows that focal loss helps long-tail ICD coding from discharge summaries, but it is text-only and uses one global focusing strength. Cross Modal Focal Loss (CMFL) is the closest existing work. For RGB-plus-depth face anti-spoofing, it suppresses one channel's loss when the other channel is confident, precisely so that a channel blind to an attack is not forced to fit it.

We borrow that intuition about blindness but change two things. First, CMFL's gate is the other modality's current confidence. With rare clinical positives, that rule reproduces starvation, because it switches off vitals learning exactly when the notes encoder catches a death. Our gate is instead an estimate of whether this modality itself can learn the patient. Second, we combine the gate with class-rarity weighting, which CMFL does not consider. CMAF-Net also couples fusion with a class-balanced adaptive focal loss, but its two branches are both image backbones on pathology patches. Its gains therefore say nothing about text-versus-time-series competition.

The work that constrains the design concerns calibration. Focal loss is known to distort calibration (AdaFocal, Dual Focal Loss, Calibrating Deep Neural Networks using Focal Loss). When One Modality Sabotages the Others shows that overconfident wrong unimodal predictions override correct ones. Any per-modality focal scheme could make that sabotage worse, so we measure both effects rather than assume them away.

The Objective

Let $p^m_i$ be the probability that modality $m$'s auxiliary head assigns to patient $i$'s true label, and $p^f_i$ the same quantity for the fused head. Let $\alpha_{y_i}$ be a class weight, inverse to the prevalence of patient $i$'s label, $\gamma$ the focusing strength, and $q^m_i$ a cross-fitted visibility score defined below. The training loss is

$$ \mathcal{L}_i = \alpha_{y_i}\Big[(1-p^f_i)^{\gamma}\,(-\log p^f_i) \;+\; \lambda \sum_m (1-p^m_i)^{\gamma}\,\min\!\big(1, q^m_i/\tau\big)\,(-\log p^m_i)\Big], $$

where $\lambda$ weights the unimodal terms and $\tau$ is a visibility threshold. Below $\tau$, a modality's push on patient $i$ is scaled down in proportion to $q^m_i$.

The visibility score $q^m_i$ is computed as follows. We split the training set into $K$ folds. On each held-out fold, we fit a lightweight probe (logistic regression on frozen encoder features) using only modality $m$, trained on the other folds. Its held-out probability of the true label is $q^m_i$. Because each patient is scored by a probe that never saw that patient, a low $q^m_i$ indicates that similar patients are not separable from this modality. A low score from a model that has simply memorized the patient cannot occur. We refresh the scores once mid-training using the current encoders' features, again cross-fitted. We also report a variant that keeps them fixed, so any instability from moving targets is visible.

Research Program

The architecture is held fixed so that every comparison isolates the loss. Each note is encoded with Bio+Clinical BERT, as in Deznabi et al. Multiple notes are pooled by learned attention rather than a uniform mean. The 17 vitals are encoded by an LSTM. A fused head sits on concatenated representations, and two auxiliary unimodal heads are attached.

The program has three stages. The first is diagnostic. On MIMIC-III in-hospital mortality, we train with fused-only focal loss and record each encoder's linear-probe AUPRC on positives across training. We compare these curves against the same encoder trained alone. A persistent gap for the vitals encoder confined to the minority class is the starvation effect we claim. If no gap appears, the premise fails and we report that.

The second stage is controlled. We build a semi-synthetic version of the dataset with known blindness by replacing the vitals of a chosen fraction of positive stays with the vitals of matched negative stays. These positives are then provably invisible to the vitals encoder. This lets us check directly whether the gate's $q^m_i$ recovers the injected set, and whether ungated per-modality focal loss over-fits it while the gated version does not.

The third stage evaluates on real tasks: MIMIC-III mortality, MIMIC-III and eICU-CRD early sepsis following Wang et al.'s windows, and MIMIC-IV-MM mortality and 30-day readmission.

Baselines

The naive reference point is the fixed architecture trained with unweighted cross-entropy on the fused head, matching how Deznabi et al. trained. Deznabi et al. report an AUC of 0.899 for their multimodal model on MIMIC-III in-hospital mortality, 0.875 for text only, and 0.835 for time series only. We reproduce their split (11,579 / 2,570 / 2,573 stays) and must land near 0.899 before any comparison counts.

The strongest standard imbalance baseline is the same model with class-weighted fused focal loss, the direct multimodal transfer of MARN's recipe. It is the method a practitioner would reach for first.

The strongest multimodal-balancing competitor adds per-modality auxiliary heads to that fused focal loss, each trained with its own focal loss and no gate. A second version substitutes CMFL's rule of suppressing one modality's loss when the other is confident. These are the two designs our gate sits between, so they double as the key ablation. Removing the gate gives per-modality focal loss, and replacing our visibility score with the other modality's confidence gives CMFL.

The separate-training baseline trains each modality alone with class-weighted focal loss and combines them by a stacked logistic regression on held-out predictions. It cannot suffer starvation by construction. On MIMIC-IV-MM, Multimodal Risk Prediction... report an AUROC of 0.823 for late fusion on in-hospital mortality and 0.588 for notes-plus-EHR late fusion on readmission. These are the published reference points for that dataset.

For external context we cite EMERGE. It reports a mortality AUROC of 86.25 against a best baseline of 85.97, and 80.61 on MIMIC-IV 30-day readmission. Its retrieval-augmented LLM pipeline is not loss-comparable, so it serves only as context. We found no reported AUPRC on Lacuna for multimodal MIMIC in-hospital mortality, so our primary metric has no published anchor and is compared only within our own runs.

All variants share data splits, encoder initialization, and optimizer. Each receives the same tuning budget of 24 trials over learning rate, $\gamma$, $\lambda$, and class weight, with $\tau$ counted inside our method's budget. Every variant is selected on validation AUPRC, and results are averaged over five seeds.

Beating a baseline means a gain in test AUPRC whose 95% bootstrap interval excludes zero. In addition, expected calibration error must be no worse after the same post-hoc temperature scaling is applied to every method.

Why not just…?

Why not just use class-weighted cross-entropy or oversample the rare patients? This only reweights by class, so every positive is pushed equally hard regardless of which modality could explain it. That leaves the starvation dynamic intact. Oversampling also invites memorization of the few minority stays, as Adrian Tousignant et al. caution. Both methods appear as baselines. If either matches the gated loss, the extra machinery is not justified and we will say so.

Why not just train each modality separately with focal loss and stack them? This is the real threat. It removes starvation entirely, and late fusion was the best mortality configuration in Multimodal Risk Prediction.... The case for joint training rests on the fused head learning interactions, such as vitals that matter only given a note's context. Separate training cannot learn those. The diagnostic stage tells us whether there is anything to gain. If separately trained encoders already match the best jointly trained encoder on minority-class probes, and stacking matches the gated model, the conclusion is to stack. That would still be a useful clinical recommendation, and we would report it as the main result.

Why not just bolt an existing modality-balancing method such as CAT-GS onto fused focal loss? CAT-GS reallocates gradient across encoders at the batch level, based on how confident each encoder is overall. It cannot tell a rare positive the vitals could learn from one they are blind to. Its 86.29% accuracy on CREMA-D was obtained with balanced labels. The CMFL and ungated per-modality arms test the two simplest batch-free versions of this idea. The semi-synthetic blindness experiment tests directly whether sample-level visibility matters. If confidence-based suppression handles the injected blind positives as well as our gate, the cross-fitting is unnecessary.

Proposed Experiments and Decisive Checks

The primary metric is test AUPRC. Secondary metrics are AUROC, sensitivity at 90% specificity, and expected calibration error after temperature scaling. We also report a sabotage rate, adapted from Zhang et al.: the fraction of fused errors where one unimodal head was confidently wrong and the other was right.

Four results would decide the question. First, the starvation gap must appear in stage one. Second, in the semi-synthetic test, the gate must recover the injected blind positives (we report the AUROC of $-q^m_i$ for identifying them). Ungated per-modality focal loss must then lose AUPRC as the injected fraction grows, while the gated loss stays flat.

Third, on real MIMIC-III mortality and sepsis, the gated loss must beat fused focal loss, ungated per-modality focal loss, and stacked separate training on AUPRC without a higher sabotage rate. Fourth, as a robustness check, we drop notes at test time. This mirrors CMFL's finding that its RGB-only model fell to 12.0% error rate versus 15.4% for a standard multi-head model. The gated model's vitals head should retain more minority-class sensitivity than the fused-focal model's, because its vitals encoder was not starved.

If the third check fails while the first two hold, the contribution narrows to a diagnosis and a controlled demonstration rather than a deployable loss, and the note will be written up that way.