When the Survey Answer Style Becomes the Signal: Response-Style Leakage in Tree Models of Social Need

Clinical prediction models are starting to use patient-reported surveys about social circumstances (money worries, food insecurity, transportation, social support) instead of neighborhood averages linked by zip code. Many of these questions are Likert items, where the patient picks a point on a graded scale such as 1 = "strongly disagree" to 5 = "strongly agree". People differ in how they use such scales. Some agree with nearly everything. Some favor the extremes. Some stay near the middle. These habits, called response styles, are known to vary with culture, language, and education. Tree ensembles, which are averages of many decision trees that each split a feature at a threshold such as "item ≤ 2.5", cannot tell a patient who truly faces severe hardship from one who simply answers every question at the extreme. So a model can quietly learn answering habits that differ by demographic group. The result can look like a social-risk effect while actually producing group-dependent miscalibration and explanations that do not hold up from one subgroup to the next.

This note proposes to measure that leakage directly and to test a fix. The fix estimates each respondent's response style from their answers across unrelated survey items, then feeds the tree ensemble scale-adjusted latent scores instead of raw item codes. The test bed is the All of Us cohort, where XGBoost on self-reported social data already predicts healthy aging well. The question is whether correcting how Likert answers are measured makes the model's probabilities and explanations equally trustworthy across groups, at no cost in discrimination.

Core Claim

We claim that tree ensembles trained on raw Likert social-need items absorb respondent response style as a proxy feature, and that this produces two measurable harms. The first is calibration gaps between demographic groups. The second is unstable explanations: split thresholds and feature rankings that shift across bootstrap resamples and across subgroups. A psychometric correction applied before the trees, which estimates per-person scale shifts and returns adjusted latent scores, should shrink both harms while leaving overall AUROC roughly unchanged. The claim is falsifiable. If response style does not differ across groups in the data, or if correcting for it changes neither subgroup calibration nor explanation stability, the hypothesis fails, and we will report that.

Why This Is Not Already Done

The closest work is Social Determinants of Healthy Aging: An Investigation using the All of Us Cohort. It trains XGBoost on All of Us survey and EHR data, explains it with TreeSHAP, and reports parity ratios. It feeds survey answers in as given and never asks whether the ordinal codes mean the same thing to different respondents. That is the variable we isolate.

Two other papers motivate the measurement focus. Accounting for racial bias and social determinants of health in a model of hypertension control shows that tree ensembles on non-clinical features carry large racial error gaps, and it repairs them only afterward with race-specific decision thresholds. We ask whether part of such gaps originates upstream, in measurement. Its survey is binary, so Likert response style cannot be studied there. Evaluating the Impact of Social Determinants on Health Prediction finds that social features act on calibration and subgroup error rather than AUROC. It uses only zip-code aggregates and names individual-level measurement as its main limitation. Its finding tells us to judge the correction on calibration and parity, not AUROC.

On the measurement side, the Likert literature in the file establishes the scale construction and the ordinal-versus-interval tension (Shaohua Guo; Ong et al.), as well as composite construct scoring (Liu et al.). It stays inside behavioral research and never meets a clinical tree model. On the interpretability side, rule distillation from forests (PECLIDES Neuro) and exact tree counterfactuals (Desiderata for Interpretability) supply our explanation-quality probes. Neither has been run on survey-derived social features, where a threshold of "≤ 2" carries meaning only if the scale means the same thing across people.

Research Program

The program has three parts.

The first part is diagnosis. From the Likert-type items in the All of Us social-determinants and lifestyle survey modules, we select items from content domains that are deliberately unrelated to one another. For each respondent we compute standard response-style indices: mean agreement across these heterogeneous items (acquiescence), the share of endpoint answers (extremity), and the share of midpoint answers. We then test whether these indices differ across race, ethnicity, preferred language, and education. We also test whether, once true construct level is held fixed, the indices predict how a respondent answers individual social-need items. Psychometricians call this differential item functioning. Finally, we measure how much of a raw-item XGBoost model's attribution flows through features that are mostly response style.

The second part is the correction. We fit an ordinal item-response model, meaning a model that treats each answer as a noisy reading of a hidden trait. In this model each respondent has a latent level on each social-need construct, plus person-level shifts for acquiescence and extremity, estimated from the heterogeneous items. The ensemble then receives posterior latent construct scores, with response-style indices withheld, together with the same clinical and demographic features used by Kim et al.. Where the substantive direction is clear, we add monotonic constraints on these scores, so that greater hardship can never lower predicted risk.

The third part is explanation-quality evaluation. Using the correction from the second part, we fit both raw and corrected models and compare their explanations along three lines. We extract rules in the PECLIDES style and record how rule-set size trades against accuracy. We measure how stable split thresholds and TreeSHAP rankings are across bootstrap refits and across subgroups. On a distilled single tree, we run the Sokol–Flach counterfactual search and check whether the suggested changes to social items mean the same thing across groups. This third part is secondary; the second part carries the main claim.

Baselines

The naive reference is XGBoost on raw integer item codes, configured as in Kim et al., with random over-sampling and the same healthy-aging outcome and cohort definition. The reported number to match is AUROC 0.793 (95% CI 0.788–0.796) on the age-50+ cohort, with accuracy and positive-predictive-value parity ratios between 0.86 and 1.06. We will re-run this pipeline ourselves to obtain subgroup calibration error, which that paper does not report. Beating it means a lower worst-group expected calibration error and a smaller between-group calibration gap, with AUROC inside its confidence interval.

The strongest classical competitor is the psychometrician's default: average each construct's items into a composite score and train the same XGBoost on the composites. This is also the most dangerous baseline, because averaging already reduces item-level noise.

Two ablations isolate the key ingredient, the person-level response-style adjustment. The first is the same item-response model without person shifts. The second is a cheap version that ipsatizes, meaning it centers and scales each respondent's answers by their own mean and spread across all items.

We also include the post-hoc repair of Hu et al., group-specific decision thresholds applied to the raw model. We further include area-level indices linked by zip code as in Yang et al., and L1-penalized logistic regression on corrected scores, as the transparent reference. All models share the same train/validation/test splits, the same hyperparameter search budget (an equal number of trials per model family), and the same feature set apart from the survey representation.

Why not just…?

Why not just average items into composite scores? Averaging removes item-specific noise but not acquiescence. A yea-sayer shifts every item in the same direction, and that shift survives in the average. If composites match the corrected model on subgroup calibration, then response style is not the operative problem, the simpler pipeline wins, and we will say so. This is the comparison the study is built to settle.

Why not just fix subgroup errors afterward with group-specific thresholds? Hu et al. show that threshold adjustment can equalize false-negative rates. It leaves the probabilities, feature attributions, and rules unchanged, however, so clinicians would still see explanations built on answering habits. It also requires conditioning decisions explicitly on protected attributes. We will apply it to both raw and corrected models. The proposal adds value only if the corrected model needs smaller adjustments, or produces explanations that are more stable across groups.

Why not just use zip-code deprivation indices and skip surveys? Yang et al. report that area-level features leave AUROC unchanged and, through transit-usage proxies, raise false-positive rates for some vulnerable groups. Running both representations on the same All of Us patients shows directly whether individual answers, once corrected, carry information that area averages lack.

Why not just ipsatize? Ipsatization is the real low-cost threat. It removes both genuine general hardship and response style, since a person who truly struggles in every domain looks the same as a yea-sayer. The item-response model separates the two using items from unrelated content domains. The semi-synthetic experiment below tests whether that separation actually works.

Proposed Experiments

The first experiment is a semi-synthetic recovery test. On real All of Us answer patterns, we inject a known construct-to-outcome effect and group-dependent response styles of known size. We then check which representation (raw codes, composites, ipsatized answers, or the full model) recovers the true effect and keeps subgroup calibration equal. This is the only setting with ground truth, and it decides whether the method can work at all.

The second experiment is the main real-data comparison: healthy aging in the age-50+ and age-85+ cohorts, where Kim et al. report AUROC 0.862 for the older group. Metrics are AUROC, overall and worst-group expected calibration error, false-positive and false-negative rate gaps, and parity ratios. The decisive result is a reduced worst-group calibration error for the corrected model relative to both composites and ipsatization, with AUROC unchanged.

The third experiment measures explanation stability across 50 bootstrap refits. We record the spread of chosen split thresholds on each social construct, the rank correlation of TreeSHAP importances between subgroups, and the accuracy of PECLIDES-pruned rule sets at a fixed rule budget. For comparison, PECLIDES Neuro reports 94.7% accuracy at 724 rules against 97% at 1,447 rules on ADNI. That is a different domain, cited only as a reference for the pruning trade-off.

A negative result in the second experiment, combined with a positive one in the first, would mean response style is real but too small in All of Us to matter, which is still a useful bound for anyone deploying survey-based social-risk models. One further note on scope: for MIMIC-IV, the benchmark used by Yang et al., no Likert-based social features were found on Lacuna, so that benchmark is out of scope.