Hospitals increasingly have several kinds of data on each patient at once: notes, chest X-rays, ECGs, labs. The labels they care about, such as diagnosis codes, follow a steep power law. A few hundred codes appear constantly, and thousands appear fewer than ten times. A popular recipe for small or messy clinical data is the "hybrid" classifier. A pretrained neural network turns raw inputs into fixed embedding vectors, and a classical model such as logistic regression or gradient-boosted trees makes the decision. This recipe is often better calibrated and less prone to overfitting than end-to-end training. It breaks down at the rare end of a power law, though. The standard practice trains one classical model per label, and a label with four positive examples gets a model fit to four points.
We propose to replace thousands of per-label heads with one gradient-boosted tree ensemble. It scores (patient, label) pairs, using features of the patient drawn from each modality together with features describing the label: its position in the code hierarchy and which codes it tends to co-occur with. A rare code then reuses splits learned from its frequent relatives. Because the input features are grouped by modality and the model is a single tree ensemble, exact tree-based attributions answer "which modality drove this code" in milliseconds rather than the hour a sampling explainer needs.
Core Claim
The claim has two parts. First, a label-conditioned tree ensemble should achieve higher rare-code F1 than per-label hybrid heads. Here "rare" means codes with 2–10 training examples, the slice where Rare Codes Count reports that CAML and DR-CAML score 0.00 micro-F1. The ensemble works on frozen multimodal embeddings, and the gain should come without hurting frequent codes. Second, the same model yields per-modality attributions that are exact with respect to the model, fast enough for bedside use, and that expose "modality competition" as a measurable quantity per code. Modality competition is the effect where adding a data stream makes predictions worse.
Formally, let $\phi_m(x)$ be the feature vector the system extracts from modality $m$ of patient $x$. Let $\psi(\ell)$ be a fixed descriptor of label $\ell$: its ancestors in the ICD tree plus a low-dimensional embedding of its co-occurrence statistics. The model is
$$ s(x,\ell) = F\big(\phi_{\text{notes}}(x), \phi_{\text{CXR}}(x), \phi_{\text{ECG}}(x), \phi_{\text{labs}}(x), \psi(\ell)\big), $$
where $F$ is one boosted tree ensemble trained with a logistic loss on positive and sampled negative pairs, and $s$ is the score that label $\ell$ applies to patient $x$. Trees can split on $\psi$ and then on patient features. This lets them learn rules such as "within the sepsis subtree, lactate-trend features matter." Such rules transfer to children of that subtree that have almost no examples.
Positioning Against Prior Work
The motivating evidence comes from three places. MC-BEC already feeds ClinicalBERT, RadBERT and waveform-transformer embeddings into LightGBM for emergency-care prediction. It evaluates only binary and three-class tasks, gives no interpretability analysis, and reports that adding labs and radiology lowered decompensation performance. We adopt its hybrid architecture and extend it to thousands of labels. We also turn its modality-competition observation into something measured for each code.
The dental hybrid study (Reliable Dental Radiograph Diagnosis via Calibrated Hybrid Representation Learning) shows frozen ConvNeXt embeddings with logistic regression reaching 99.1% accuracy and ECE below 0.009. That task has four balanced classes and one modality, so it tells us nothing about the tail. Impact of metadata in multimodal classification of bone tumours motivates the tail problem directly: its image-plus-metadata model reaches 69.7% overall accuracy but 0% on Ewing sarcoma, which has 6 cases.
The long-tail coding literature supplies our label descriptors, but it ends in neural heads. HiCu uses the ICD hierarchy for a coarse-to-fine curriculum and hyperbolic label embeddings. Rare Codes Count builds label representations with GCNs over co-occurrence graphs and notes their memory cost. We reuse their structural signals only as fixed inputs $\psi(\ell)$ to the tree model, so there are no graph layers at inference.
DILA is the closest work on the interpretability side. It inserts a sparse autoencoder, a model that rewrites a dense vector as a few active "dictionary" features, before a linear ICD head on notes. This gives instantaneous explanations and a MIMIC-III macro-F1 of 27.2 versus 26.5 for PLM-ICD. It is text-only, uses a linear head, and names features with a 70B LLM. We borrow its per-modality sparse-dictionary idea as an optional feature extractor. We differ in three ways: several modalities, a shared nonlinear head, and grouping attributions by modality rather than naming features with an LLM.
Two papers constrain what we claim. On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods found that TreeSHAP and LIME slowed fraud analysts by 14–16 seconds without improving decisions. We therefore claim only faithfulness and latency, not better clinician decisions. The replicability audit (Automated Medical Coding on MIMIC-III and MIMIC-IV) shows that tuning per-code decision thresholds rivals architectural change. Every model here gets tuned thresholds, and macro-F1 is computed as the plain mean of per-class F1.
Research Program
We freeze pretrained encoders for notes (a clinical language model), chest X-rays and ECGs, and compute simple summary statistics for labs. Each modality's embedding is reduced to a few hundred features. The main variant uses a sparse autoencoder trained per modality, following DILA. The ablation uses PCA. Missing modalities stay missing: gradient-boosted trees route missing values natively, and we additionally drop modalities at random during training.
Label descriptors $\psi(\ell)$ combine the one-hot ancestors of the code at each ICD level with a 32-dimensional spectral embedding of the co-occurrence matrix computed on training data only. Training pairs are all positives plus hard negatives. Hard negatives are sampled from the same ICD chapter and from the top of a cheap retriever's ranking.
Scoring all 8,000+ codes per patient with trees would be slow, so inference uses a shortlist. A linear retriever built on the same features proposes the top few hundred candidate codes, and the ensemble re-scores only those. Retriever recall is measured and reported separately, so that shortlist misses are not confused with head errors.
Explanations come from exact TreeSHAP on the single ensemble, run only for predicted codes. Attributions are summed within each modality group. This gives a per-code, per-patient vector with one entry per modality. Averaging that vector over patients gives each code a modality profile. A code whose profile is concentrated on a modality that, when ablated, improves that code's F1 is our operational definition of modality competition.
Baselines
The naive reference points come in two forms. One is predicting each code at its training-set prevalence with a tuned threshold. The other is PLM-ICD on notes alone with per-code threshold tuning, following the replicability audit, which reports that tuned thresholds let simple models compete. Any method that does not clearly beat tuned PLM-ICD on the rare slice has no case.
The strongest existing text-only systems are DILA, with a reported MIMIC-III macro-F1 of 27.2 (PLM-ICD 26.5), and Rare Codes Count, with a reported 6.72 micro-F1 on codes seen 2–10 times (7.15 with augmentation), where CAML and DR-CAML report 0.00. We rerun both on the "MIMIC-III clean" split with the corrected macro-F1 formula. We do not compare to their published numbers, because the audit shows historical macro-F1 figures were miscomputed.
For the multimodal MIMIC-IV setting (notes linked to MIMIC-CXR-JPG and MIMIC-IV-ECG), there is no reported result found on Lacuna for multimodal ICD-10 coding, so we construct the baselines ourselves. The first is the MC-BEC recipe applied per label: one binary LightGBM per code, on the same frozen features. This is the direct competitor for the hybrid question. The second is RadFusion-style late fusion, which averages per-modality probabilities; RadFusion reports 89.0% accuracy for this on PE, beating either modality alone. The third is an end-to-end fine-tuned multimodal model trained with asymmetric loss, the loss HiCu uses to down-weight easy negatives.
The key ablation isolates label sharing from the choice of tree model. It is a shared bilinear model that scores $\phi(x)^\top W \psi(\ell)$ on identical features and descriptors; it is equally sharing and instantly explainable. A second ablation removes $\psi(\ell)$ from the ensemble, leaving only a label ID.
All methods use the same frozen features where applicable, the same patient-level splits, the same hyperparameter search budget of 50 trials on validation, and per-code threshold tuning on validation. Beating the field means two things. The first is higher rare-slice (2–10 examples) micro- and macro-F1 than every baseline, with 95% bootstrap intervals over patients that exclude zero. The second is frequent-code (Top-50) macro-F1 within one point of the best baseline.
Why not just…?
Why not just train per-label logistic regression on frozen embeddings, as in the dental study? It is the simplest hybrid and instantly explainable. But a per-label model with four positives and hundreds of features is exactly the setting where every per-label method on MIMIC-III collapses, as the CAML and DR-CAML zeros in Rare Codes Count show. The honest threat is not per-label linear models but the shared bilinear ablation. If it matches the tree ensemble, then the contribution is label conditioning, not trees, and we will report it that way.
Why not just fine-tune a multimodal model end-to-end with asymmetric loss and tuned thresholds? The replicability audit makes this a real contender on frequent codes. However, the same audit reports that all models still struggle on rare codes after threshold tuning. MC-BEC and the bone-tumour study show that joint training can let one stream drown out others and can zero out rare classes. It is included as a baseline under the same threshold tuning. If it wins on the rare slice, the proposal fails.
Why not just run TreeSHAP on MC-BEC-style per-label LightGBMs? Exact tree attributions are already cheap per model. The problem is elsewhere: per-label models are weak on the tail, and attributions computed over raw embedding coordinates cannot be read clinically. Grouping by modality fixes the second problem for either design. Our latency experiment times both, so if per-label TreeSHAP is equally fast, latency is not a differentiator and we will say so.
Why not just retrieve similar rare cases and prompt a multimodal LLM, as SGG-ICL and MedPromptX do? SGG-ICL improves long-tail F1 by 6.9% on surgical scene graphs, but at 3–5× latency (about 5 seconds per rare sample). MedPromptX needs a Med-Flamingo pass per query. Neither yields attributions that are exact with respect to the model. We treat retrieval as a latency-accuracy comparison point rather than a baseline tuned under equal compute.
Proposed Experiments and Decisive Checks
MIMIC-III clean, text only. This establishes comparability with DILA and Rare Codes Count before modalities are added. The decisive check is whether the label-conditioned ensemble on sparse text features lifts the 2–10-example slice above rerun Rare Codes Count, and whether it beats the bilinear ablation.
MIMIC-IV ICD-10, multimodal cohort. This is restricted to admissions with linked notes, chest X-ray and ECG. Codes are stratified by training frequency into bins (1, 2–10, 11–100, >100). The decisive check is whether, compared with per-label LightGBM, the gain concentrates in the rare bins and grows when modalities are added. The second check is modality competition: for each code, drop one modality at a time at test time, and test whether the attribution profile predicts which ablations help. We report the rank correlation between each modality's attribution share and the drop in F1 when that modality is removed.
Bone tumour dataset. On the dataset from Impact of metadata in multimodal classification of bone tumours, descriptors are the WHO category (benign, intermediate, malignant) plus metadata-based class priors. The check is whether Ewing sarcoma recall rises above zero under patient-grouped cross-validation. With 6 cases, we report exact counts and treat this only as a stress test.
Faithfulness and latency. Faithfulness is measured by deletion: removing the top-attributed modality group should reduce the score more than removing a random group. Latency is end-to-end wall-clock per admission on one CPU core, split into retrieval, scoring and explanation. It is compared against KernelSHAP on the same model; DILA reports KernelSHAP takes over an hour per note in its setting. Calibration is measured by ECE on the rare bins, since the dental study's calibration advantage for hybrid heads is untested in the tail.
Failure conditions. The proposal fails if any of three things happens: the bilinear ablation matches the ensemble, end-to-end asymmetric-loss training wins on rare bins, or attribution shares do not predict modality-ablation effects. Each outcome is informative and will be reported as such.
References
- Rare Codes Count: Mining Inter-code Relations for Long-tail Clinical Text Classification
- Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
- Reliable Dental Radiograph Diagnosis via Calibrated Hybrid Representation Learning
- Impact of metadata in multimodal classification of bone tumours
- HiCu: Leveraging Hierarchy for Curriculum Learning in Automated ICD Coding
- DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction
- On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods
- Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
- RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHR
- Modeling Long-Tail Relations in the Operating Room via In-Context Multimodal Learning
- MedPromptX: Grounded Multimodal Prompting for Chest X-Ray Diagnosis