When a model reads a patient's ICU record, it sees two very different things: a stream of vital signs sampled at irregular times, and a pile of free-text clinical notes. Current multimodal models encode the notes with dense text encoders and let the two streams interact through cross-modal attention, a mechanism in which features from one stream ask "which parts of the other stream are relevant to me?" and pull those parts in. A long line of text-classification work points to a problem with this recipe. How often a term is written, and how long a document is, mostly reflect writing style rather than content. Throwing counts away and keeping only whether a term appeared (a binary presence representation) repeatedly beats richer weighting in sentiment, topic, and clinical text. In the ICU this matters more than usual, because note length and repetition track who wrote the note and how busy the unit was. A dense text branch can use those cues as a shortcut, learn faster than the vitals branch, and crowd it out.
We propose to use the vitals as the questions and a timestamped set of binary concept-presence indicators from the notes as the answers. At each time window, the vital-sign features attend over the set of clinical concepts that have been mentioned so far. Each concept is represented only by its identity and the window in which it first appeared, never by its count or its position in the prose. The central claim is that this representation removes length and repetition shortcuts by construction, and that it keeps the vitals modality in use. We also claim that query-dependent attention over the presence set gives accuracy gains beyond simply adding the binary vector to each time step. Both claims are testable on MIMIC-III, and both can fail.
Core Claim and What Is New
The mechanism is continuous queries attending to binary keys. Let $h_t$ be the vitals encoder's feature at time window $t$. Let $S_t$ be the set of concepts whose first non-negated mention occurs at or before $t$, with a learned embedding $e_c$ per concept $c$ and a learned embedding $\tau_{t_c}$ for the window of first mention. Then
$$z_t = \sum_{c \in S_t} \operatorname{softmax}_{c \in S_t}\!\left(\frac{(W_q h_t)^\top W_k (e_c + \tau_{t_c})}{\sqrt{d}}\right) W_v (e_c + \tau_{t_c}),$$
and the fused feature $[h_t; z_t]$ feeds a recurrent predictor. Here $W_q, W_k, W_v$ are projection matrices and $d$ is the key dimension. Repeating a sentence, lengthening a note, or rephrasing the same finding leaves $S_t$ unchanged, so $z_t$ is exactly invariant to these edits. A dense note encoder carries no such guarantee. The vitals stay continuous on purpose. The file flags loss of magnitude as the main weakness of binarization, and a heart rate of 140 versus 80 is signal rather than noise. We binarize only the modality where frequency is known to be mostly noise. Negation is handled by treating a negated mention ("no chest pain") as a distinct concept, since the plain presence of "chest pain" would otherwise be misleading.
The closest prior work either keeps text dense or makes attention sparse rather than the representation binary. CTPD fuses irregular vitals with clinical notes on MIMIC-III through shared temporal prototypes and Slot Attention, but it passes notes through a deep continuous text encoder. We replace that text side with a presence set and use CTPD as our strongest comparison. Kim and Shin use Sparsemax to force cross-attention weights to exactly zero and a gate to preserve modality identity. Their inputs remain dense embeddings, so their sparsity lives in the attention weights, whereas ours lives in what the memory contains. FMS and SAWGAN-BDCMA treat "binary" only as the output label and attend over continuous features. Zhang et al. voxelize binary event-camera polarities into continuous grids before attention. These papers show that nobody has yet put binary indicators inside the attention memory.
The binary-representation side supplies the motivation and the method. Wang and Manning show that binarized counts improve Naive Bayes, and that support vector machines on log-count ratios of those counts (NBSVM) reach 91.22% on IMDB, ahead of more complex neural models. Amiri et al., Stepanov et al. and Ениколопов et al. find binary lexical indicators competitive or more stable than frequencies in clinical and mental-health text. Liu et al. show that binary word occurrence works as a noise-robust bridge between text and another modality.
The modality-dominance literature supplies the failure mode we target. Wang et al. (2020) show that modalities overfit at different rates, and that naive joint training on Kinetics (71.7%) can underperform the best single modality (72.6%). Tamim et al. document gate thrashing and gradient conflict at the fusion layer.
Research Program
We will build a concept-extraction pipeline over MIMIC-III notes. It maps note text to a fixed vocabulary of lexical and phrase units, with negation-marked variants and a first-mention timestamp aligned to the vitals windows. From this we produce $S_t$ for each window. The vitals encoder is held identical across all variants, as is the downstream predictor.
We then compare text-side representations while holding everything else fixed. The variants are the dense encoder CTPD uses, count-valued keys where each concept key is scaled by its mention count, and binary presence keys.
The analysis has three parts. First, we measure how fast each branch overfits using the overfitting-to-generalization ratio of Wang et al. (2020). Our prediction is that the binary text branch overfits more slowly than the dense one, and closer to the vitals branch. Second, we measure reliance on the vitals by the drop in test performance when vitals are replaced with their training mean. Third, we run a counterfactual repetition test. We duplicate existing sentences or pad notes with templated boilerplate from the same corpus, then measure prediction shift. The binary model's shift is zero by construction, so the informative quantity is how large the shift is for the dense and count models. A secondary study probes the threshold sensitivity noted in the file. We compare strict presence with the median-based binarization of Solorio et al., which sets a concept to 1 only if its count exceeds the training median.
Baselines
The naive reference point is the vitals-only model with the same encoder and predictor. Any multimodal variant must beat it, and Wang et al.'s Kinetics result warns that some will not.
The strongest existing method is CTPD. On 48-hour in-hospital mortality it reports a 1.89% F1 improvement over the best of MedFuse and DrFuse. On 24-hour phenotyping it reports a 1.2% AUROC gain and a 1.92% AUPR gain. No absolute CTPD F1, AUROC or AUPR values were found on Lacuna, so we will re-run CTPD's released configuration on our identical cohort splits and report its numbers alongside ours. We will give it the same hyperparameter-search budget and five seeds. Beating it means a higher mean in-hospital-mortality F1 and phenotyping AUPR, outside the seed-level standard deviation. Failing that, it means matching it while being invariant under the repetition test and showing smaller vitals-ablation drops, which is a robustness claim rather than an accuracy claim, and we will label it as such.
The ablation that isolates the key ingredient keeps our exact architecture but swaps binary keys for count-weighted keys. If counts do as well, binarization is not doing the work. A second ablation replaces query-dependent attention with uniform weights over $S_t$, which reduces to adding the mean concept embedding at each time step. If this matches full attention, the cross-modal attention adds nothing.
We also include late fusion of a vitals model with NBSVM on binarized note unigrams and bigrams, following Wang and Manning. Its fusion weight is tuned on validation data.
Why not just…?
Why not just late-fuse NBSVM on binary note features with a vitals model? This is the most serious threat. It captures the length-invariance benefit with no attention at all, and the file's evidence says such linear baselines are hard to beat on text. Its limitation is that it sees the whole stay's notes at once and cannot tie a mention to the physiological window it explains. If our model does not beat it on phenotyping AUPR, we will conclude that binarization is the whole story and the attention is overhead. The late-fusion baseline above is run specifically to settle this.
Why not just add the binary concept vector to each time step (early fusion)? A linear projection of a binary vector is exactly a sum of concept embeddings, so this is the uniform-weight ablation. The file offers no direct evidence either way for this setting, and our honest prior is that it will be close. Attention earns its place only if the gap is measurable. The largest gap is expected for patients with many concurrent mentions, where the relevant concept depends on the current vitals, so we report results stratified by $|S_t|$.
Why not keep the dense encoder and fix dominance with gradient balancing such as G-Blend or CAT-GS? Wang et al. (2020) lift Kinetics to 75.8% this way, and Tamim et al. reach 86.29% on CREMA-D. However, CAT-GS depends on unimodal teachers, and Tamim et al. note that on noisy large-scale data, input noise outweighs optimization control. Balancing gradients also does not remove a length shortcut; it only changes how fast the model learns it. We run dense encoder plus G-Blend as a baseline, and the repetition test will show whether it remains sensitive to note padding. The two remedies are compatible, so we also report binary keys combined with G-Blend.
Why not just truncate or length-normalize notes? Truncation removes late information, and normalization rescales counts without removing repetition within the kept text. The count-key ablation with per-note normalization tests this directly.
Proposed Experiments
All experiments use MIMIC-III v1.4 with CTPD's two tasks: 48-hour in-hospital mortality, scored by F1, AUROC and AUPR, and 24-hour classification of 25 phenotypes, scored by macro AUROC and AUPR. Every model is trained with five seeds on identical splits under a matched tuning budget, and we report parameter count and per-record inference time against CTPD's reported 9.4M parameters and 5.42 ms.
The decisive checks are four comparisons against the baselines above. Binary keys must beat count keys, or the representation claim fails. Full attention must beat uniform weighting, or the fusion claim fails. The binary model must beat NBSVM late fusion, or attention is unnecessary. On the repetition test, and when vitals are removed, the binary model must show smaller prediction shift than the dense model, and it must also rely more on vitals, or the dominance story is wrong. We will report every outcome, including a result in which binary presence helps only through late fusion. That result would still tell practitioners that the text side of clinical fusion models should be simplified rather than enlarged.
References
- CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Fine-grained Multiple Supervisory Network for Multi-modal Manipulation Detecting and Grounding
- SAWGAN-BDCMA: A Self-Attention Wasserstein GAN and Bidirectional Cross-Modal Attention Framework for Multimodal Emotion Recognition
- Object Tracking by Jointly Exploiting Frame and Event Domain
- Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
- The UMD CLPsych 2016 Shared Task System: Text Representation for Predicting Triage of Forum Posts about Mental Health
- Depression Severity Estimation from Multiple Modalities
- Features of texts written by a frustrated person
- Less is More: Zero-Shot Learning from Online Textual Documents with Noise Suppression
- What Makes Training Multi-Modal Classification Networks Hard?
- CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery
- Domain Adaptation for Authorship Attribution: Improved Structural Correspondence Learning