Whose Money, and Was It There? Pre-Training Language Models to Read Material Buffers in Hardship Narratives

Research on socioeconomic resilience keeps returning to one finding: material resources such as income, insurance, and a pension act as shock absorbers that decide whether stress turns into depression, burnout, or suicide. Most of the evidence about these buffers in real clinical and death records sits in free text, not in structured fields. A social worker writes "patient denies financial difficulty; daughter pays rent," or a coroner writes "recently retired, no pension, lost insurance two weeks prior." To use such text at scale, a model has to answer three small questions correctly. Whose resource is it (the patient's or a relative's)? Is it present or absent? Is it a lifelong condition or a recent shock? Current clinical language models get the headline label right and the three questions wrong. On the MIMIC-SBDH benchmark, Bio-ClinicalBERT reaches F1 of 0.96 on one social category but fails 44.5% of negation tests and 70.9% of attribution tests (MIMIC-SBDH).

We propose to continue pre-training a long-context clinical encoder on hardship narratives, meaning clinical social-history sections, social work notes, and death-investigation narratives. During that pre-training the model gets a label-free objective that forces it to track holders, polarity, and timing. We then test two things. First, whether this fixes the whose/whether/when failures better than the cheaper remedy of adding counterfactual examples during fine-tuning. Second, whether the fix changes the epidemiological conclusions that resilience researchers draw from model-extracted variables. In two sentences: domain text alone does not teach a model whose resource a sentence describes or whether it is missing, so we build those distinctions into pre-training on unlabeled hardship text. We judge success by robustness tests and by whether downstream estimates of income-buffer effects move toward the ones computed from gold labels.

Core Claim

The errors that matter for resilience research are errors of holder, polarity, and timing. They are exactly the errors that separate a family with a working buffer from the "Dual Adversity" profile in which both caregiver and patient lack resources (Cluster analysis of family resilience in adolescents with emotional disorders). Caregiver studies show the same thing: burden depends on the household's income rather than any one person's (caregiver burden in moyamoya disease).

We claim three things. Plain domain continued pre-training will improve topic detection but will only partly reduce these errors. A cue-targeted pre-training objective, run on hundreds of thousands of unlabeled notes, will reduce them further than fine-tuning-time augmentation can with a few thousand labeled documents. And the residual errors are not random noise: they bias estimated associations between material buffers and outcomes, and they may bias them differently across demographic groups.

Relation to Prior Work

Three papers motivate the gap by showing that current SDOH extractors are built on backbones never adapted to social-history language. Mezuk et al. fine-tuned general RoBERTa on 1,291 NVDRS sentences to find retirement transitions (F1 0.92). They showed that retirement mentions in suicide narratives rose by a factor of 1.495 during the Great Recession, but they measured mentions, not whether a pension or savings buffer was present. Lybarger et al. built SHAC, a corpus of annotated social-history sections, and a sentence-level event extractor on Bio+Discharge Summary BERT. The multi-stage NVDRS pipeline chains retrieval, verification, and temporal extraction using general LLMs and FLAN-T5. It beats fine-tuned BioBERT by 17.7% F1 on rare stressors, yet reasoning models still show weak recall on infrequent factors. None of these papers changes the backbone's representations. We change the backbone and reuse their tasks as evaluations.

MIMIC-SBDH supplies both the failure evidence and the method we reuse for measuring it: CheckList behavioral tests covering negation, attribution, and history. Ji et al. supply the architectural recipe. Their continued pre-training of Longformer to 4,096 tokens lifted UMD F1 from 59.74 at 512 tokens to 72.47. Their corpus, however, was Reddit, and their targets were diagnoses. We keep the long-context recipe, replace the corpus with institutional hardship narratives, and change the objective.

LDA on 0.95 million UCSF social work notes shows that such notes contain rich financial and support content missing from structured fields. It also shows that bag-of-words methods cannot handle negation or attribution, which is where we start.

Understanding Intrinsic Socioeconomic Biases in Large Language Models constrains the design. Larger general models tie demographic identity more strongly to poverty; for example, Llama 2's Poverty Association Ratio is 0.885 for "Divorced Indigenous mother" versus 0.29 for "White man." An extractor that guesses a buffer from identity cues instead of from the text would therefore produce differential misclassification. Our evaluation must check error rates by group, not only overall.

Research Program

The pre-training corpus combines social-history and social work sections from MIMIC-III (credentialed access) and, subject to data-use approval, NVDRS narratives outside the evaluation years. Clinical social work notes of the UCSF kind will be added only if access is obtained. We treat that source as optional, and the main results will not depend on it.

The backbone is a 4,096-token Longformer. We initialize it two ways: from the general Longformer checkpoint, and from MentalLongformer. The second initialization tests whether mental-health pre-training helps or hurts on institutional text.

The objective has two parts. The first is selective masking. We use a small lexicon plus a dependency parse to find holder cues (possessives and kinship heads such as "her son's," "patient's wife"), polarity cues ("denies," "no," "lost," "unable to," "without"), and temporal cues ("recently," "two weeks prior," "since childhood"). These tokens are masked at several times the base rate, and the rest of the text follows standard masked language modeling.

The second part is replaced-cue detection. In a fraction of sequences we automatically swap one holder cue for another kinship term, flip one polarity cue, or change one temporal cue. The model predicts, token by token, whether each cue was swapped. This needs no labels. It forces the representation of a resource mention to depend on its holder, polarity, and timing across the whole document, not just within the sentence. That matters because a narrative often names the daughter in one paragraph and the rent in the next.

We will build the swap generator to produce fluent edits and will audit a sample by hand for implausible swaps.

Baselines

The naive reference point is Bio-ClinicalBERT fine-tuned directly on each task, exactly as in MIMIC-SBDH. The reported CheckList failure rates are 44.5% for Community negation and 70.9% for Education attribution, and topic F1 reaches 0.96 for Community-Present. We will rerun it under our fine-tuning protocol rather than rely on those numbers alone.

The strongest existing extraction system on SHAC is fine-tuned mSpERT. Ramachandran et al. report overall F1 of 0.861 for it on the n2c2/UW SDOH challenge. It is the number to beat on event extraction. Our encoder will be dropped into the same span-and-relation extraction head so that only the backbone differs. Beating it means matching or exceeding 0.861 overall and clearly improving on argument-linking and status arguments, the places where attribution and negation errors appear.

On NVDRS retirement detection, the reference is general RoBERTa at F1 0.92 from Mezuk et al.. For long narratives, MentalLongformer is the strongest long-context domain model. Its reported F1 is 72.47 on UMD and 76.32 on CLPsych15 (Ji et al.), but no reported result was found on Lacuna for MentalLongformer on SHAC, MIMIC-SBDH, or NVDRS. We will therefore run it ourselves.

The key ablation is plain continued pre-training on the identical corpus, with the same tokens, steps, and initialization but uniform masking and no replaced-cue detection. This isolates the objective from the domain text.

All encoder variants get the same number of pre-training tokens where applicable. Fine-tuning uses the same labeled splits and the same hyperparameter search budget, with a fixed number of trials per model. We report means over five seeds.

Why not just…?

Why not just fine-tune Bio-ClinicalBERT with counterfactual negation and attribution examples? This is the real threat. Rewriting labeled sentences ("denies" to "reports," "patient's" to "his wife's") and flipping their labels may close most of the CheckList gap cheaply. We therefore run a two-by-two design: pre-training objective (plain versus cue-targeted) crossed with fine-tuning augmentation (none versus counterfactual). The augmentation budget is matched to the size of the CheckList-style perturbation families, and the templates used for augmentation are held out from those used for testing.

If augmentation on the plain backbone matches the cue-targeted backbone on held-out perturbation types and on naturally occurring attribution errors in real notes, we will report that the pre-training objective is unnecessary. The expected advantage of pre-training is generalization to phrasings and long-range holder references that the templates never show. For that reason, the decisive test set is naturally occurring cases, not templates.

Why not just prompt a frontier or reasoning LLM? One-shot GPT-4 reached 0.652 and 0.625 F1 against mSpERT's 0.861 on SHAC (Ramachandran et al.). DeepSeek-R1 showed weak recall on rare stressors in NVDRS (multi-stage pipeline). Those results predate current models, however, so we will include a many-shot prompted open-weights model run on the same evaluation. NVDRS and MIMIC data-use rules generally bar sending records to proprietary APIs, which is also a practical reason a locally trainable encoder matters.

Why not just LoRA-tune a general open model? The evidence that base pre-training dominates the fine-tuning recipe comes from code generation (PEFT comparison for Python). It may not transfer to this domain, so we will not rely on it. We will include a LoRA-tuned open decoder model trained on the same labeled data as an additional baseline. If it matches our encoder on robustness, the case for a specialized backbone weakens to one about cost and deployability.

Proposed Experiments

Behavioral robustness. We extend the MIMIC-SBDH CheckList suites to Economics and Environment, adding perturbations specific to buffers (insurance lost versus kept, pension held by spouse versus decedent). We report failure rates next to topic F1.

Natural error set. We will hand-annotate roughly 1,000 naturally occurring spans from SHAC and MIMIC-SBDH that involve a non-patient holder or a negated resource. For each span we record holder, polarity, and timing. This measures whether the gains survive outside templates.

Standard benchmarks. We run SHAC event extraction against mSpERT and the NVDRS retirement task against RoBERTa. We also run a document-level NVDRS subtask that we define and annotate on a held-out year: acute (within two weeks) versus chronic financial stressor, and decedent versus family holder. The two-week window follows the multi-stage pipeline.

Downstream consequence check. This check decides whether the work matters for resilience research. On an annotated NVDRS subset with gold labels for financial-buffer mentions, we estimate the association between recession period and retirement-related suicide mentions, conditioning on the presence of a financial buffer. We compute this once from gold labels and once from each model's extractions. A model that extracts well by F1 but whose estimates diverge from the gold-label estimates has failed the test that matters.

We also stratify misclassification rates by recorded sex, race, and age. Because of the bias audit, we expect identity-linked guessing in larger general models. A successful result requires both smaller divergence from gold-label estimates and no widening of the gap in error rates between groups.

Falsification. The proposal is falsified if the cue-targeted objective does not beat plain continued pre-training on the natural error set, or if counterfactual augmentation on a plain backbone matches it.