Social-media sentiment models learned to read emotion without human labels by deleting the emoji a writer attached to a post and training a network to guess it back. Clinical notes carry a similar marker that the author writes for free. Nearly every hospital discharge summary ends with a discharge medication list and a clinician-written list of diagnoses. We propose to treat these sections the way emoji were treated. The model hides them, reads the rest of the note, and predicts the whole set of hidden items jointly. The set is predicted together because a patient's drugs reflect a patient's diseases: metformin with lisinopril and furosemide points to diabetes, hypertension and fluid overload all at once. Learning to recover that set should push a text encoder to represent the patient's full comorbidity profile, meaning the group of diseases that co-occur in one person, rather than one primary diagnosis. It should do this without any billing codes.
The opportunity matters because automated coding of clinical text is not mainly limited by model size or document length. A careful replication of six coding systems found that document length barely affects error, and that the real barrier is data sparsity in the long tail of rare codes (Automated Medical Coding on MIMIC-III and MIMIC-IV). Weak supervision for clinical notes is still limited to single-label or entity-level tasks. Comorbidity structure, where it is modeled at all, is added afterwards on top of a fully supervised model. A self-supervised target built from clinician-written markers addresses all three gaps with one mechanism.
Core Claim
Predicting the hidden discharge medication list and the hidden free-text diagnosis list from the remaining note text is a distant supervision signal for multi-label clinical coding. It is free, abundant, multi-label by construction, and noisy in the way emoji are noisy. One drug serves many indications, much as one emoji serves both sincere and sarcastic posts. We claim that an encoder pretrained this way gives two benefits when fine-tuned for ICD coding. ICD codes are the International Classification of Diseases codes used for billing and statistics. The first benefit is better recall on rare codes and in low-label regimes than an encoder pretrained only with masked language modeling. The second is that the gain depends on two things. The marker set must be predicted jointly with a co-occurrence-aware head, not as independent yes/no outputs. And the markers must be removed from the input, so the model cannot copy them.
The method comes from emoji distant supervision. DeepMoji pretrained on 1.2 billion tweets by predicting which of 64 emoji had been removed and then transferred the encoder to emotion and sarcasm tasks. Sentiment-specific word embeddings showed that emoticon labels reshape a vector space so that words sharing contexts but opposite in polarity, such as "amazing" and "terrible", stop sitting together. The clinical analogue of that geometric failure is "chest pain" versus "denies chest pain". Purely distributional training places these phrases close together, but they imply different medication lists. We borrow the recipe, not the domain: tweet-trained affect models transfer poorly to clinical prose, as the research file's discussion of domain shift notes.
Research Program
The pretraining corpus is the full set of MIMIC-III and MIMIC-IV discharge summaries. A section parser, building on the section-identification work behind MedSecId, locates the discharge medication and discharge diagnosis sections. Those sections are then removed from the input entirely, along with any admission medication list. That way the target cannot leak through a near-duplicate section. Medication strings are normalized to ingredient level, and the most frequent few hundred to one thousand ingredients form the marker vocabulary, which plays the role of DeepMoji's 64 emoji. Free-text diagnosis lines are kept as a second marker vocabulary after light normalization of frequent phrases. These lines are written by the treating clinician rather than a billing coder, which makes them a different and partly independent view of the same patient.
The encoder is a biomedical pretrained language model with the segment pooling and per-label attention of PLM-ICD. Segment pooling splits long notes into 512-token chunks and pools them. Per-label attention gives each output its own weighting over the note. Using this encoder lets downstream comparisons isolate the pretraining signal rather than architecture.
The comorbidity component is a label-cloze head trained jointly with the encoder. At each training step we sample a random subset of the true marker set and reveal it to the head. The head then predicts the remaining markers given both the note and the revealed ones. This is the masked self-attention reranker of Tsai et al. moved inside training and conditioned on text. Their reranker scores candidate code sets from co-occurrence alone, after a frozen text model has proposed them. Ours sends co-occurrence gradients back into the encoder. At inference the head decodes iteratively: it starts from an empty revealed set, commits the most confident predictions, and repeats. The same head is reused when fine-tuning on ICD codes, so the coding model can also condition on codes it has already committed to. Because no tool can recover a code it never proposes, this design targets the failure Tsai et al. leave open: a rare true code missing from the top-$k$ candidate pool.
Three analyses accompany the main results. The first is a negation probe. On sentences containing negated versus affirmed findings, we measure whether marker pretraining separates the two in representation space more than masked language modeling does. This is the clinical counterpart of the antonym separation that sentiment-specific embeddings achieved. The second quantifies label noise. For each normalized drug, we measure how many distinct ICD chapters it co-occurs with, then test whether the drugs that transfer best are the indication-specific ones. The third is a cross-note-type transfer check, described below. It asks whether the representation helps on notes that never contain the markers.
Baselines
The naive reference point is the same encoder initialized from its public biomedical checkpoint and fine-tuned directly on ICD codes, which amounts to PLM-ICD as published. This is also the strongest existing supervised method in the file. PLM-ICD reports micro-F1 of 59.8% on MIMIC-III full-code prediction. For historical context, CAML reports micro-F1 0.54 and precision@8 0.71 on the same task. The replication study shows that macro-F1 values in older papers were miscomputed, so that correcting the formula roughly doubles them. The file also gives no reported result on Lacuna for PLM-ICD macro-F1 on MIMIC-III clean or MIMIC-IV. We will therefore rerun PLM-ICD ourselves under the replication study's corrected metrics, stratified splits and tuned decision thresholds, rather than compare against published numbers. Beating it means a statistically significant gain in corrected macro-F1 and in recall on codes with fewer than about 50 training occurrences, with micro-F1 no worse than parity.
The second baseline is domain-adaptive masked language modeling on the identical corpus for the identical number of steps, with the marker sections masked as ordinary tokens. This is the ablation that matters most. It spends the same compute on the same text, so any advantage of our method must come from predicting markers as a structured set rather than from extra in-domain reading.
The third baseline is the reranker of Tsai et al., applied post hoc on top of the PLM-ICD baseline. It reports 1% to 16% relative macro-F1 improvement in the all-code setting across base models. It is the strongest existing way to inject comorbidity structure, so it tests whether end-to-end joint modeling adds anything over a decoupled reranker. We also stack it on our model to check that the gains are not redundant.
For the zero-label regime, the baseline is KeyClass. It builds keyword labeling functions from ICD chapter descriptions and reports F1 0.625 on 19 top-level MIMIC-III chapters, against 0.678 for the supervised FasTag. We compare in its single-label setting, using our model's top chapter. We also extend its labeling functions to multi-label output and compare in that setting.
Two more ablations isolate the key ingredients. In the first, the marker set is predicted with independent sigmoid outputs, with no cloze head. In the second, marker sections are left in the input, which tests the shortcut and should collapse transfer. All systems share the encoder, data splits, a fixed hyperparameter search budget of equal size, and three seeds.
Why not just…?
Why not just pretrain on the ICD codes themselves, since MIMIC has them? On frequent codes in MIMIC this alternative is a real threat and will probably match or beat us. ICD labels are the target, and any proxy pays a noise cost. The proposal's value lies where codes are scarce: rare codes, small labeled fractions, and institutions whose coding practice differs. We run this alternative as an upper-bound comparator at matched compute. We will report honestly if marker pretraining only matches it on rare codes. In that case the claim narrows to label-free settings.
Why not just feed the medication list in as an input feature? For discharge-time coding this is cheap and likely strong, and we include it as a baseline. It cannot help where the list does not exist yet, such as admission-day physician and nursing notes. The cross-note-type experiment targets exactly that setting. If marker pretraining gives no gain there over masked language modeling, the representation-learning claim fails.
Why not just extend KeyClass to multi-label? KeyClass's authors identify multi-label extension as open and report that the method struggles on categories present in fewer than 1% of records. Description keywords are sparse precisely for the rare codes. We build the multi-label extension ourselves, so this is measured, not assumed.
Why not just add the Tsai reranker to PLM-ICD? It is cheap, but it operates on a frozen candidate pool and cannot recover a code the text model never proposes. Our experiments report candidate-pool recall for rare codes, so this limit is visible directly.
Proposed Experiments
The main experiment is ICD coding on MIMIC-III clean and MIMIC-IV, using the replication study's splits and corrected metrics. Fine-tuning uses 1%, 10% and 100% of labeled admissions. We report micro-F1, corrected macro-F1, precision@8, and recall bucketed by code frequency.
The decisive check is the rare-code bucket at the 10% label fraction. If marker pretraining with the cloze head does not beat masked language modeling at matched compute on that bucket, the core claim is false.
The second decisive check is the pair of independent-output versus cloze ablations. If the gain is the same without the joint head, the comorbidity component is unnecessary, and we would report the method as plain proxy pretraining.
The third check is cross-note-type transfer. We predict the admission's codes from first-24-hour physician and nursing notes, which contain no discharge markers, against the masked language modeling and code-pretraining baselines.
Finally, we run zero-label chapter classification against KeyClass's 0.625 F1. Here a model trained only on marker prediction maps to ICD chapters through drug–chapter co-occurrence learned on an unlabeled split. This tests whether the hidden-marker signal alone is a usable form of distant supervision.
References
- Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
- Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm
- Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification
- A New Public Corpus for Clinical Section Identification: MedSecId
- PLM-ICD: Automatic ICD Coding with Pretrained Language Models
- Modeling Diagnostic Label Correlation for Automatic ICD Coding
- Explainable Prediction of Medical Codes from Clinical Text
- Classifying Unstructured Clinical Notes via Automatic Weak Supervision