Explaining the Update, Not the Score: Trust in Continuously Updated Clinical Risk

Hospitals are replacing admission-time risk scores such as APACHE-II and SOFA with models that recompute a patient's risk every hour, or every minute in the operating room. These dynamic models predict better. Their explanations, however, are still built for one-off decisions: at each update, a tool such as SHAP (which splits a prediction into per-feature contributions) explains the current score from scratch. At a real bedside, a clinician watching those per-feature bar charts reshuffle from hour to hour may be watching the "erratic narrator" that made radiologists trust an explainable model less than a black box. We propose to explain each change in risk instead: which newly arrived measurements moved the score, and by how much. By construction, today's explanation is then yesterday's explanation plus a small, checkable increment. It also points the clinician at exactly the new raw data worth auditing.

The two-sentence version is this. The trust backfire documented for static AI explanations should get worse when a model updates continuously, because the explanation keeps changing its story about the same patient. Explaining the update rather than the current score removes that instability without hiding the model's reasoning, and it should improve clinicians' ability to follow the model when it is right and override it when it is wrong.

Core Claim

The explainability trust paradox is the repeated finding that exposing an algorithm's reasoning can erode trust or distort reliance rather than calibrate it. The evidence comes almost entirely from single, static decisions.

Ewals et al. ran a factorial study with 20 radiologists and found that explainable nodule characteristics lowered psychological trust relative to a black box (P=.02). Radiologists attributed the drop to inconsistent explanations across similar-looking nodules. Kunar et al. found that telling mammography readers a detector's hit rate made them less vigilant and lowered their sensitivity. Erlei et al. found that explanations led bargainers to consult the advice more often but follow it less. Expert disillusionment on seeing a system's shortcuts is reported by Pipek et al., and by Oppold et al. for data pipelines.

These papers motivate our hypothesis. None of them involves a prediction that updates over time.

Our specific claim is that in continuous monitoring, the dominant driver of the trust backfire is within-patient explanation instability across updates, not the presence of explanations as such. An explanation defined on the update removes this driver. We define the "update explanation" as an exact attribution of the risk change $\Delta r_t = r_t - r_{t-1}$ to the observations that arrived between the two updates. Compared with standard per-snapshot attribution, and with smoothed per-snapshot attribution, it should produce better-calibrated reliance: more agreement when the model is right and more overrides when it is wrong.

The mechanism for the second half of the claim comes from De-Arteaga, Fogliato and Chouldechova. In a natural experiment where the Allegheny child welfare tool displayed erroneous scores, caseworkers caught the errors because they audited the raw case history rather than the score. An update explanation is a pointer into the raw data ("the score rose 0.08 because of this lactate and this blood pressure reading"). It should make that auditing cheap under time pressure.

Research Program

The technical object. For a sequence model $f$ that maps a patient's observation history up to time $t$ to a risk $r_t$, suppose the model reads events in order without explicit timestamps. Reed et al. found that such order-only GRUs match timestamp-encoded ones on six MIMIC-IV tasks. For these models, the risk changes between updates only because new observations $N_t = \{o_1,\dots,o_k\}$ arrived. We attribute the change with exact Shapley values over those $k$ new observations, holding the history fixed:

$$\Delta r_t = r_t - r_{t-1} = \sum_{j=1}^{k} \phi_j(t),$$

where $\phi_j(t)$ is the average marginal effect of adding observation $o_j$ to the history across all orderings of the new observations. Because $k$ is small per update (typically a handful of labs and vitals per hour), exact computation over $2^k$ subsets is cheap. This avoids the sampling noise that the Attribution Stability Indicator paper identifies as a source of unstable time-series attributions.

For models that do encode elapsed time, we add one extra player, the passage of time, whose contribution absorbs any change not due to new data. A large value for this player is itself diagnostic. It is the update-level analogue of the primacy bias that Kim, Weissman and Wong expose with PaX. PaX swaps positions to separate a measurement's value ("what") from its timing ("when") in a post-hoc explanation of a single prediction. Our decomposition is complementary: it separates "new evidence" from "reinterpretation of old evidence" across successive predictions. Unlike PaX, it is designed to be displayed live and summed over a stay.

The interface. The display shows the risk curve. Each step is annotated with its top contributing new observations, and each annotation is linked to that observation's raw trend. Summing the steps recovers a stable cumulative explanation, so clinicians who want "why is risk high now" still get an answer. That answer changes only when new data arrive.

Backbones. We use backbones from the dynamic-risk literature so the explanations sit on credible models. The first is a multitask LSTM on MIMIC-III following Loreaux et al., in the spirit of Yu et al.'s real-time LSTM. Its intervention heads let us also show whether a risk drop reflects anticipated treatment. The second is an order-only GRU and a time-encoded GRU on the same data, following Reed et al. The third is a sepsis model on the PhysioNet 2019 challenge data, the setting in which PaX was developed.

Baselines

Naive reference. The simplest reference is the dynamic risk score shown alone, with raw data available on request. This is the black-box arm that Ewals et al. found was trusted more. Beating it on trust questionnaires is not enough. Our explanation must beat it on appropriate reliance, meaning a higher override rate on injected erroneous trajectories without a lower agreement rate on correct ones. A static-score arm (SOFA shown daily) anchors how clinicians behave with the tools they already know. Loreaux et al. report a SOFA mortality AUROC of 0.69 against 0.82 for their multitask LSTM on MIMIC-III, so any human-plus-model gain must be read against that model gap.

Strongest existing explainer. This is per-snapshot attribution recomputed at every update, in two versions. The first is standard SHAP. The second, the harder competitor, is PaX, which removes the primacy-bias artifacts that make vanilla attribution misleading on sequence models. On MC-MED, Kim et al. report that PaX-Feature raised insertion AUC by up to 0.03 over feature-only baselines. We found no reported result on Lacuna for PaX on PhysioNet 2019, so we will run it ourselves. All explainers receive the same backbone, the same patients and the same compute budget. Sampling-based methods get enough samples that their faithfulness saturates.

Beating these baselines computationally means matching their faithfulness to the model's change in prediction, measured by insertion and deletion over newly arrived observations, while achieving lower cross-update instability. We measure instability as the rate at which the top-5 cumulative features change between consecutive hours, and as ASI computed on clinical data. Beating them behaviorally means better appropriate reliance in the human study.

Ablation isolating the key ingredient. The ablation is per-snapshot SHAP smoothed with an exponential moving average, tuned so that its instability matches that of the update explanation. It delivers consistency without the "explain the change, point at the new data" structure. If it performs as well behaviorally, the contribution reduces to "stability matters," which is still a finding about the paradox but not a case for our method. A second ablation shows the update explanation without links to raw trends. It tests whether the benefit runs through De-Arteaga-style auditing.

Why not just…?

Why not just smooth ordinary attributions over time? This is the most serious threat. Smoothing makes explanations stable, but it does so by lagging and blurring. At the moment risk jumps, the displayed reasons still largely describe the previous hours. An update explanation attributes the jump immediately and exactly to what caused it. Whether that difference matters to clinicians is an empirical question, and the instability-matched smoothing arm settles it directly. We commit in advance to reporting a null here as evidence that consistency, not the change-based form, drives the effect.

Why not just show the raw trajectory and no attributions? De-Arteaga et al. suggest experts detect errors from raw data. But their caseworkers audited one intake record without time pressure. An ICU patient generates dozens of measurements per hour, and a trajectory display without guidance about which of them moved the model leaves the search to the clinician. Our score-plus-raw-data arm is exactly this alternative. The raw-link ablation tells us whether the attribution adds anything beyond making raw data prominent.

Why not just deploy the black-box dynamic model, since explanations lower trust? Lower trust is not the same as worse decisions. Ewals et al. saw trust fall while decision confidence rose, and Kunar et al. saw a transparency cue degrade accuracy through overtrust. Loreaux et al. argue that an unexplained low score can hide an expected intervention, and clinicians' liability concerns about unauditable models are documented in Bridging the Clinical Interpretability Gap. Our primary outcome is therefore reliance measured against ground truth, not self-reported trust. The black-box arm is a first-class competitor that could win.

Why not just use PaX live? PaX is the right tool for asking whether a model is leaning on position artifacts. But it still explains each snapshot independently, so it inherits the cross-update instability that we hypothesize drives distrust. It also highlights factors outside the guidelines, such as pH and BUN, which the skepticism findings of Erlei and Pipek suggest may provoke rejection. Our decomposition is compatible with PaX. It can be applied to a model that PaX has shown to be free of primacy bias, and the elapsed-time player gives a live flag when it is not.

Proposed Experiments

Decisive check 1: is the premise true? We first measure whether per-snapshot explanations of real dynamic models are actually unstable across hours. We compute the top-5 turnover rate and clinical-data ASI for SHAP and PaX on the MIMIC-III LSTM and on the PhysioNet sepsis model. We stratify patients by whether their risk changed materially over the interval, since instability is only a problem when the score is roughly flat. If snapshot explanations turn out to be stable when risk is stable, Gap 2's premise fails and we stop before the human study. The same analysis runs on the INSPIRE intraoperative data used by Sohrabi et al.. There, minute-level updates should make instability worst, and the trajectory signal Sohrabi et al. find in patients who look low-risk before surgery (an AUROC gain of 0.223 in the lowest static-risk tertile) makes explanations for unexpected alarms especially consequential.

Decisive check 2: does the decomposition behave as claimed? We verify the following. For order-only models the elapsed-time term is zero, as it should be by construction. For time-encoded models the term is nonzero and correlates with PaX's "when" scores. The update explanation's faithfulness on new observations is at least that of PaX.

Decisive check 3: the human study. We will recruit ICU physicians and nurses, aiming for several times the 20 readers of Ewals et al. Using a within-subject design, clinicians replay de-identified MIMIC-III trajectories hour by hour under a per-update time limit, deciding at each step whether to escalate care. The display arms are:

  • score only;
  • score plus snapshot SHAP;
  • score plus snapshot PaX;
  • score plus smoothed SHAP;
  • score plus update explanation, with and without raw links.

In a pre-registered subset of trajectories we inject a model error mid-stay. The corrupted model either inflates risk or suppresses it for a stretch of hours, mimicking the Allegheny glitch.

Outcome measures are as follows. The primary outcome is appropriate reliance: agreement with correct model outputs and override of corrupted ones, analyzed with a signal-detection model as in Kunar et al. Secondary outcomes are time to detect the injected error, how often clinicians open the raw data, and repeated trust ratings. The repeated ratings let us classify trust trajectories into the growth, arc and collapse patterns described in The Temporal Dynamics of Human AI Trust.

The proposal succeeds if the update explanation beats snapshot PaX on appropriate reliance, and also beats instability-matched smoothing. It fails informatively if smoothing ties it, which would show that stability rather than the change-based form drives the effect, or if the black-box arm wins, which would extend the paradox to dynamic monitoring.