Epidemiology suggests that diet affects depression risk differently depending on a person's metabolic state. High sugar intake tracks with depressive symptoms only in people who are already obese (Total sugar intake is associated with higher prevalence of depressive symptoms in obese adults). Metabolically healthy obesity raises depression odds in women but not in men (The Relationship Between Obesity and Depression Is Partly Dependent on Metabolic Health Status). These studies rely on surveys and billing codes, and their authors note that eating behaviors are missing from such data. The narrative that would fill this gap, such as "drinks four sodas a day" or "skips meals, binges at night", sits in free-text clinical notes. A language model could extract it.
Extraction errors are not harmless, though. Suppose the model misses dietary mentions more often for one sex, insurance class, or note template. Then the extracted exposure is misclassified at different rates in different groups, and any downstream estimate of "sugar × obesity → depression" is distorted. The obvious fix is to make the extractor ignore cues that mark a patient's group. That fix carries a risk: if men and women differ because their metabolism differs, a naive debiaser erases real signal. This note proposes a training-time masking method that suppresses group-specific shortcut tokens while explicitly sparing differences explained by metabolic status. It will be judged on whether extracted variables reproduce known effect modification without group-dependent error.
Core Claim
The method uses token attributions from Integrated Gradients, a method that scores each input word by accumulating the model's gradient along a path from a blank input to the real one. During fine-tuning, it finds words whose importance to the extractor differs between patient groups even after conditioning on the true label and on the patient's metabolic phenotype. Those words are masked more often. We claim this gives smaller subgroup gaps in dietary-exposure recall than frequency-based masking. We also claim it does so without suppressing the legitimate dependence of dietary mentions on metabolic context. The metabolic conditioning is the new ingredient: it turns "remove group-correlated attribution" into "remove group-correlated attribution that biology does not explain."
The closest prior work is Generalizing through Forgetting. It masked the 200 most frequent symptom triggers 80% of the time while fine-tuning a clinical BERT extractor, and cross-hospital F1 rose from 77.2 to 79.9. That list is chosen by frequency alone. It cannot tell a site-specific template phrase from a real descriptor, and it never looks at what the model actually relies on. Our method replaces the frequency list with a list chosen from attributions, and it conditions that choice on metabolic state.
How Prior Work Shapes the Design
Several papers supply the method. An Eye Opener Regarding Task-Based Text Gradient Saliency compared gradient saliency in fine-tuned encoders with human reading gaze. It found that saliency tracks gaze (Spearman up to 0.53 on relation extraction) while raw attention weights do not. MIMIC-IF used remove-and-retrain tests on MIMIC-IV models and showed that attention weights are unreliable explanations. It also found that models assign different importance to demographic features across subgroups. That second finding is the kind of attribution disparity our score targets, though MIMIC-IF only audited it after training, on tabular data. For the language-specific form of the attribution we use Discretized Integrated Gradients, which handles the problem that word embeddings are discrete points rather than a continuous space.
Other papers motivate the problem. The NHANES mediation analysis (Association among obesity, insulin resistance, and depressive symptoms) and the inpatient-sample study establish that metabolic phenotype modifies depression risk. That study's four phenotypes are metabolically healthy or unhealthy, each with or without obesity. It reports odds ratios of 1.442 for unhealthy obesity versus 1.066 for healthy obesity, and the sex split noted above. Both papers rely on coded or surveyed exposures.
Some papers constrain the design. FairReFuse removes confounding by gender through backdoor adjustment in audio-visual depression models. Multimodal Gender Fairness in Depression Prediction treats sex and country as direct sources of bias. Neither considers metabolic status, and both use audio and video rather than notes. Our design borrows FairReFuse's idea of conditioning on the right variables, but applies it in attribution space and adds metabolic phenotype to the conditioning set. FairerOPTH routes retinal diagnosis through 67 explicit pathological signs and reports fairness gains without lost accuracy. That result suggests grounding in clinical intermediates can avoid a "fairness tax", and we test whether this carries over to text.
Research Program
The first step is building the task, since no dietary-extraction benchmark appears on Lacuna. From MIMIC-IV discharge and progress notes we will sample about 2,000 notes, stratified by sex, recorded race and ethnicity, insurance type, and metabolic phenotype. Metabolic phenotype is derived from structured data: body mass index, fasting glucose or HbA1c, lipid labs, and ICD codes, following the inpatient-sample definitions. Clinicians will annotate spans for dietary exposures (sugary foods and drinks, confectionery, binge or night eating, meal skipping), for metabolic narrative, and for depressive symptoms. Each span is marked present, negated, or historical, because negation and timing are where keyword systems fail. The annotation guide follows the hybrid recipe of CheXbert. A lexicon labeler produces weak labels at scale. Expert spans are reserved for fine-tuning and testing.
The second step is the masking rule. Let $a_t(x)$ be the Integrated Gradients attribution of word type $t$ in note $x$ toward the predicted span label. Let $g$ be a binary group indicator used only during training (for example, sex or insurance class). Let $y$ be the gold label and $m$ the metabolic phenotype. The shortcut score of a word type is
$$ s_t = \sum_{y,m} w_{y,m}\,\Big|\,\mathbb{E}[a_t \mid g=1, y, m] - \mathbb{E}[a_t \mid g=0, y, m]\,\Big|, $$
where $w_{y,m}$ is the share of training notes in stratum $(y,m)$. A word gets a high score when the model leans on it differently across groups among patients who have the same label and the same metabolic state.
Every few hundred steps we recompute attributions on a held-out training slice. We then mask the top-$K$ words by $s_t$, using the same 80% rate and the same $K$ as the frequency baseline, so that only the choice of words differs. The ablation drops $m$ from the conditioning set.
The third step is a downstream epidemiological check. Each extractor's outputs, at the patient level, are fed into the same logistic model of depression on sugar exposure, obesity, and their interaction, with standard covariates. We compare the interaction estimate obtained from each extractor against the estimate obtained from gold annotations.
Baselines
The naive reference is a Bio+Clinical BERT span extractor fine-tuned on our labels with no masking. It shows the size of the subgroup recall gap that any method must close.
The strongest existing method is frequency masking from Generalizing through Forgetting: mask the $K$ most frequent exposure and symptom triggers 80% of the time. On its own COVID-to-oncology benchmark it reports cross-domain F1 of 79.9, up from 77.2, and recall of 72.6, up from 67.5. Those figures come from a different task, and no result on dietary extraction was found on Lacuna. We will rerun it on our data with the same backbone, the same $K$ grid, the same number of fine-tuning steps, and the same hyperparameter search budget.
A random-masking control uses the same $K$ and rate. It separates the benefit of any masking from the benefit of choosing the right words.
For large language models, the baseline is the four-step self-verification pipeline of Self-Verification Improves Few-Shot Clinical Information Extraction. Its steps are extraction, omission recovery, evidence quoting, and pruning. It raised MIMIC-IV ICD-10 F1 from 0.487 to 0.533. We will run it with the same few-shot examples drawn from our training split.
A lexicon extractor with negation rules is the transparent floor. It is motivated by CheXbert's comparison, where the rule labeler scored F1 0.743 against 0.798 for the trained model.
Two ablations isolate the key ingredient. The first is our method with $m$ removed from the conditioning, which tests whether metabolic conditioning matters. The second scrubs explicit demographic words such as "male", "African American", and "Medicaid", which tests whether shortcuts live only in explicit tokens.
Beating the strongest baseline means two things together. First, a smaller worst-group gap in exposure recall than frequency masking, with confidence intervals from patient-level bootstrap that exclude zero difference, and overall F1 no lower. Second, a downstream sugar × obesity interaction estimate closer to the gold-annotation estimate.
Why not just…?
Why not just keep frequency masking, which already works? This is the most serious threat. Frequent triggers may well be most of the shortcut problem. Frequency masking, however, cannot spare a frequent word that is genuinely informative. It also cannot catch a rare template phrase that marks one insurance class. The matched-$K$ comparison settles the question. If frequency masking equals our method on worst-group recall and on downstream interaction recovery, then attribution-guided selection is not worth its compute, and we will report that.
Why not just delete the demographic words? Explicit tokens are easy to scrub. MIMIC-IF, though, found attribution disparities spread across non-demographic features, and notes encode group membership indirectly through templates, insurance language, and phrasing. The scrubbing ablation measures how much of the recall gap survives once explicit words are gone. If almost none survives, the simpler fix wins.
Why not just use a self-verifying large language model? Evidence quoting gives auditable spans: in the self-verification paper, GPT-4's quoted spans overlapped expert spans 93% of the time. That paper did not measure subgroup gaps. In addition, Revisiting the MIMIC-IV Benchmark found zero-shot models far behind fine-tuned ones on note-based prediction, with AU-ROC of 0.50 to 0.61 against 0.87 to 0.88. We will include the pipeline and report its group gaps. It may win on overall F1, and if it also has small gaps, that is a useful finding.
Why not just collect more diverse notes or reweight groups? The RETFound analysis (Yukun Zhou et al.) and FairDiffusion both report that group disparities persist when data are scaled without addressing imbalance. Reweighting, in addition, cannot tell biological effect modification from shortcuts. We will run group-balanced sampling as an additional baseline under the same budget.
Decisive Checks
Two planted tests come first, because they have known answers.
The first plants a fake shortcut. We insert a neutral template phrase into notes of one group, correlated with the label. Our method should rank this phrase near the top of $s_t$ and cut the model's reliance on it. Reliance is measured by remove-and-retrain accuracy loss, as in MIMIC-IF.
The second plants genuine effect modification. We construct notes in which a sugar mention determines a depression-symptom label only when obesity is documented, and we make obesity more common in one sex. The metabolically conditioned score should leave the sugar token unmasked. The unconditioned ablation should mask it. If the conditioned and unconditioned versions behave the same here, metabolic conditioning adds nothing and the core claim fails.
On real MIMIC-IV data we then hold out each group stratum in turn and report worst-group recall and overall F1. As a check on attribution sensitivity, we repeat the masking selection with Discretized IG and with a padding-token baseline. If the top-$K$ list changes substantially between the two, the selection is unstable, and we will say so rather than report a single favorable configuration.
References
- Total sugar intake is associated with higher prevalence of depressive symptoms in obese adults
- The Relationship Between Obesity and Depression Is Partly Dependent on Metabolic Health Status: A Nationwide Inpatient Sample Database Study
- Generalizing through Forgetting - Domain Generalization for Symptom Event Extraction in Clinical Notes
- An Eye Opener Regarding Task-Based Text Gradient Saliency
- MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset
- Discretized Integrated Gradients for Explaining Language Models
- Association among obesity, insulin resistance, and depressive symptoms: a mediation analysis
- FairReFuse: Referee-Guided Fusion for Multi-Modal Causal Fairness in Depression Detection
- Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA & China
- Fairer AI in ophthalmology via implicit fairness learning for mitigating sexism and ageism
- Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT
- Self-verification improves few-shot clinical information extraction
- Revisiting the MIMIC-IV Benchmark: Experiments Using Language Models for Electronic Health Records
- A foundation model for generalizable disease detection from retinal images
- FairDiffusion: Enhancing Equity in Latent Diffusion Models via Fair Bayesian Perturbation