Temporal Shifts on Anatomy Graphs for Longitudinal Chest X-rays

When a radiologist reads a chest X-ray, they compare it with the patient's earlier films and ask, region by region, whether each finding is getting better, staying the same, or getting worse. Models that try to do this face a correspondence problem. Two films taken days apart differ in patient position, breathing depth, and exposure, so the same pixel location does not show the same tissue. The strongest current model, CheXLearner, handles this with a detector that localizes anatomical regions, an alignment step on a curved (hyperbolic) feature space, and supervision from a medical text encoder. It compares exactly two films. Cheap temporal operators from video take the opposite approach. The temporal shift module (TSM), introduced in TSM, copies a slice of each frame's feature channels into the neighboring frame. This gives a 2D network temporal reasoning at zero extra arithmetic, but it assumes pixel $(x,y)$ at time $t$ corresponds to pixel $(x,y)$ at time $t-1$, which is the very assumption longitudinal X-rays break.

The idea of this note is to do the temporal shift on the nodes of a medical knowledge graph instead of on pixels. A knowledge graph here is a network whose nodes are anatomical regions and diseases and whose typed edges encode relations such as "contains," "is adjacent to," and "commonly co-occurs with." A node named "left lower lung zone" means the same tissue in every film, so the graph supplies, for free, the correspondence that pixel grids lack. A parameter-free channel shift between visits should then be enough temporal modeling. A graph attention layer, which learns how much each neighbor should influence a node, can read these shifted features. Its edge weights then depend on the patient's trajectory: for example, it can upweight a region-to-effusion edge only when that region is worsening. The claim is falsifiable. Shifting on region nodes should match heavy pairwise alignment, while shifting on pixel feature maps should fail. Because shifts stack across layers, the model should also gain from a third or fourth prior film, which pairwise models cannot use.

Core Claim

The main obstacle to cheap longitudinal medical imaging models is correspondence, not temporal capacity. Once correspondence comes from knowledge-graph identity, the zero-cost temporal mixing that TSM, Temporal Interlacing Network, and X-ViT showed on video becomes valid for medical sequences. If the attention scores over the graph are computed from those shifted features, the influence of external knowledge becomes conditional on disease trajectory.

This contribution has three parts, and each targets a gap in prior work. First, it resolves a tension the literature leaves open. CheXLearner argues that flat feature comparison fails because of registration noise, while shift methods treat temporal alignment as a flat memory copy. We predict both are right, and that the copy fails only when it is indexed by pixels. Second, it extends progression detection from pairs to sequences. Third, it replaces the fixed, structure-only edge weighting that knowledge-graph clinical models like GAMENet use with weighting that depends on how the patient is changing over time.

Relation to Prior Work

Three lines of video work supply the temporal operator. TSM shifts a quarter of channels forward and backward in time inside residual branches. It reports that shifting all channels hurts spatial features, which is the constraint we inherit when we sweep the shift fraction. Temporal Interlacing Network learns fractional per-group offsets and shows shifting is equivalent to a regularized temporal convolution. X-ViT shifts key and value channels between frames before spatial attention. All three operate on pixel or patch grids of video, where the camera is roughly fixed. None has been tested where frames are misregistered by weeks of physiology, and we move the operator to graph nodes precisely to avoid that problem. The closest graph use, LA-SGCN, combines shift graph convolutions on a fixed skeleton with channel attention. Its graph carries no external knowledge, its edges are untyped, and its attention reweights channels rather than edges, so the attention cannot respond to motion trajectory.

The neighbor-weighting mechanism comes from Graph Attention Networks, which compute per-edge weights from the features of both endpoints. GAT reports 0.973 micro-F1 on the inductive protein-interaction benchmark versus 0.768 for GraphSAGE. Work on heterophilous graphs, Spectrum-Constrained and Skip-Enhanced Graph Fraud Detection, motivates the expectation that fixed averaging blurs nodes whose neighbors disagree, as a stable region does next to a worsening one. We reuse GAT but feed it time-mixed features, and that has not been done.

Knowledge-graph medical models motivate the knowledge component and constrain its expected size. SLAKE found that adding knowledge triplets improved knowledge-based question accuracy by 2.0% on static images. This is a modest gain, and it warns us not to expect large effects from the graph alone. GAMENet injects drug-interaction graphs through fixed-weight graph convolutions, with time handled separately by recurrent networks. AM-MRG obtains disease awareness from Hopfield memories attached to a 7B language model at large compute cost. Our model fuses time and knowledge in the same message-passing step and stays small.

Research Program

Each study in a patient's sequence passes through a shared anatomical region detector of the kind CheXLearner uses. The detector returns pooled features for a fixed set of named regions; Chest-ImaGenome annotates 12 regions and 9 disease categories. We build a typed knowledge graph over these. Region nodes are linked by anatomical containment and adjacency. Region–disease edges mark where each finding can occur. Disease–disease edges encode clinical relations such as effusion with atelectasis, mapped to canonical concepts through the Unified Medical Language System, a large standard vocabulary of medical concepts. Disease nodes start from text embeddings of their names.

At visit $t$, let $h_{i}(t) \in \mathbb{R}^{C}$ be the feature of node $i$. Before each graph layer, a causal shift replaces a fraction $\rho$ of channels with the same node's channels from the previous visit, and leaves the rest unchanged:

$$\tilde h_i(t) = \big[\, h_i^{(1:\rho C)}(t-1) \;\|\; h_i^{(\rho C+1:C)}(t) \,\big].$$

The shift is causal so the model can run online as new films arrive. It sits inside a residual branch, as in TSM, so the unshifted identity of each node survives. Graph attention then scores each typed edge from the shifted features and a relation embedding $r_{ij}$:

$$e_{ij}(t) = \mathrm{LeakyReLU}\big(a^{\top}[\,W\tilde h_i(t) \,\|\, W\tilde h_j(t) \,\|\, r_{ij}\,]\big), \qquad \alpha_{ij}(t) = \mathrm{softmax}_{j}\, e_{ij}(t),$$

Here $W$ is a learned projection, $a$ is a learned scoring vector, and the softmax runs over the neighbors $j$ of node $i$. Because $\tilde h$ mixes visits $t-1$ and $t$, the weight $\alpha_{ij}(t)$ can depend on the direction of change. Stacking $L$ layers gives a temporal reach of $L$ visits at no added arithmetic beyond the attention itself. A per-(region, disease) head predicts improved, unchanged, or worsened.

Shifts index visits rather than days, and the gaps between X-rays are irregular. This is the open limitation the file notes for shift operators and for Time-Aware Neighbor Sampling. We therefore test a gated variant that scales the shifted channels by a learned exponential decay in the time gap, in the spirit of the decay gating used in AdaptViG.

The analyses go beyond accuracy. We sweep $\rho \in \{1/8, 1/4, 1/2\}$ to see whether shifting erodes node identity, which is TSM's spatial-degradation finding transposed to ontology nodes. We also examine whether learned edge weights separate worsening from improving trajectories for the same region–disease pair.

Baselines

The naive reference point uses only the current and previous region features. It feeds their concatenation and difference to an MLP, with no graph and no further history. This measures how much of progression detection is solved once regions are localized, and every other gain must be read relative to it.

The strongest existing method is CheXLearner. It reports 81.12% average accuracy and 80.32% F1 for progression detection on the 500-case radiologist-annotated Chest-ImaGenome test set. We evaluate on the same gold set and use the same detector family so that region features are comparable. If CheXLearner can be retrained, it gets the same training pairs and the same number of hyperparameter trials as our model. Beating it means exceeding 80.32% F1 on the pairwise gold set at a fraction of its parameter count. A tie on pairs combined with a clear gain on patients with three or more prior films would also count, since that setting is outside what a pairwise model can address.

Pixel-level TSM inserts the same causal shift into the image backbone instead of on graph nodes, with everything downstream identical. This is the ablation that tests the core claim.

A full space-time transformer over all region tokens from all visits, with no knowledge graph, is the strongest generic competitor. With around a dozen regions and a few visits, full attention is cheap. Our model has to win through its inductive bias, not its efficiency.

A per-node recurrent network over visits, followed by the same graph attention, tests whether a learned temporal model beats a parameter-free shift once correspondence is fixed.

Two ablations isolate the key ingredient. The first computes attention scores from unshifted features while still passing shifted features as messages, which removes trajectory-conditioned weighting but keeps temporal information. The second replaces GAT with a fixed-weight graph convolution, in the style of GAMENet. A third control keeps the graph's degree sequence but randomizes its edges, to test whether the knowledge content matters or only the sparsity.

All variants share the detector, training data, optimizer, and tuning budget, and run on matched hardware for latency reporting.

Why not just…?

Why not just feed the difference between current and prior region features into a classifier? For two films this may be nearly as good, and the naive baseline exists to find out. Explicit differencing grows with each extra prior film and has no natural way to let knowledge-graph edges respond to change. The shift reaches $L$ visits with fixed width. The decisive evidence is the comparison on patients with three or more films. If differencing with concatenated history matches the shift there at equal parameters, the shift is only an implementation detail, and we will report the contribution as anatomy-indexed temporal mixing in general.

Why not just put TSM in the image backbone? The video evidence (TSM, X-ViT) assumes camera-stable frames. CheXLearner's finding that flat comparisons fail under positioning and breathing changes predicts that pixel shifts will mix unrelated tissue. The pixel-level TSM baseline tests this directly. If it performs as well as node-level shifting, our central claim is wrong.

Why not just run a small transformer over all region tokens, with no knowledge graph? This is the most serious threat, because the token count is small enough that efficiency is irrelevant. Our bet is that typed anatomical and clinical edges help when labels are limited and noisy, and on rare region–disease combinations. SLAKE's modest 2.0% gain from knowledge triplets warns that the advantage may be small. We will compare the two models overall, on the hilar and other overlapping regions where CheXLearner reports its largest gains, and under reduced training data. If the transformer matches us everywhere, the knowledge graph earns no place in the model.

Why not just use fixed graph convolutions, as GAMENet does? Fixed weights cannot express "this edge matters only while that region is worsening." The fixed-convolution and static-score ablations measure exactly what dynamic weighting adds.

Proposed Experiments

Training uses Chest-ImaGenome's automatically labeled region pairs, which number over 427,000. Evaluation uses the 500 radiologist-annotated cases. We report per-region and average accuracy and F1, with confidence intervals from bootstrapping over patients, since 500 cases make small differences unreliable.

For the multi-visit test, we build sequences by chaining consecutive comparisons for patients with three or more studies. Whether enough such patients appear in the gold set must be checked first; if not, we will evaluate multi-visit gains on a held-out portion of the automatically labeled data and flag that its labels are noisier. We also report parameters, FLOPs, and latency per added visit.

Three outcomes would decide the proposal. First, node-level shifting must clearly beat pixel-level shifting; this is the correspondence claim. Second, computing attention scores from shifted features must beat the static-score ablation; this is the trajectory-conditioned knowledge claim. Third, accuracy must improve when history extends from one prior film to two or more, while the pairwise result stays within reach of CheXLearner's reported 80.32% F1. If the first holds and the others fail, we still learn something useful: knowledge-graph nodes are the right place to align longitudinal images, even if dynamic weighting over them adds nothing.