Nearly every model that reads chest X-rays over time compares exactly two images, the current study and the one before it, and says whether a finding such as pleural effusion is improving, stable, or worsening. Real patients often have five, ten, or twenty prior studies, and their histories contain far more evidence than any single pair. We propose to keep the cheap pairwise model and run it on many pairs from a patient's history, not only the adjacent ones. Each pairwise judgment then becomes a noisy measurement of the difference between two unknown per-exam severity values. We recover those values with spectral initialization: we place the measurements in a matrix and take its leading eigenvector, a method that provably gives a good starting estimate for a nonconvex recovery problem. We then refine that starting point with a robust objective that ignores wildly wrong comparisons. The output is one consistent severity trajectory per finding. Any requested comparison, including the adjacent pair a benchmark asks about, is read off as a difference along that trajectory.
The reason to expect this to help is that pairwise judgments contradict each other. A model may say exam 1 to exam 2 is worsening, exam 2 to exam 3 is worsening, and exam 1 to exam 3 is improving. Those three statements cannot all be true. Methods that recover globally consistent quantities from inconsistent pairwise measurements, called synchronization, are well developed for camera poses and image correspondences. They have never been used to pool clinical judgments across a timeline.
Core Claim
Our central claim is narrow. On a held-out comparison between two exams, a robust synchronized estimate built from the model's judgments on many other pairs in the same patient's history will be more accurate than the model's direct judgment on that pair. The gain should grow with the number of available exams and with the rate of grossly wrong pairwise outputs. Those wrong outputs come, for example, from portable front-to-back versus upright back-to-front projections, or from changes in line and tube placement.
The spectral step matters for one specific reason. The robust refinement objective is nonconvex, so a poor starting point can leave it stuck. A least-squares starting point is pulled toward outliers. The eigenvector start is meant to land in the region from which the refinement reliably reaches the right answer. If an ordinary least-squares fit matches the full method, the spectral and robust machinery is not earning its keep, and we will say so.
Relation to Prior Work
The longitudinal chest X-ray literature motivates the problem.
- BioViL-T splits image features into what the current study shows and what changed since the prior study, and shows that conditioning on the prior reduces invented comparison statements.
- CoCa-CXR restricts each current-image patch to attend only to the matching neighborhood in the prior image.
- TempA-VLP adds a cross-exam encoder to separate static findings from interval change.
- DeltaNet subtracts prior features from current ones before writing the report.
All four reason over one pair, which the research file identifies as the main open limitation. They are complementary to this proposal: any of them can serve as the pairwise measurement device we synchronize. We do not compete with their architectures. We add a model-agnostic layer that uses the rest of the history.
The method comes from synchronization and spectral initialization.
- Normalized Spectral Map Synchronization builds a block matrix of noisy pairwise maps and divides each block by node degree, which keeps the eigengap stable on irregular graphs. It then takes leading eigenvectors and matches SDP recovery at roughly 100× lower runtime (2.25 s vs 203.12 s on CMU). - Dual Quaternion SE(3) Synchronization uses a power-method eigenvector as the start for a generalized power method, with linear convergence under bounded noise.
We reuse the degree normalization because clinical history graphs are highly irregular. Patients differ in exam count, and model reliability varies across pairs. We reuse the "eigenvector start, then local refinement" pattern. Our group is far simpler, scalar severity offsets rather than rigid motions or permutations, but our noise is far worse.
The robustness side constrains the design. Truncated Wirtinger Flow and Median Truncated Wirtinger Flow show that a single extreme measurement can hijack a plain spectral estimate, and that trimming high-leverage measurements before and during refinement fixes this. That evidence is why we do not trust a plain eigenvector or a plain least-squares fit as the final answer.
CRONOS is a cautionary reference. It finds that learned sequence models often lose to simply copying the last image. Our pooled estimate must likewise beat the trivial "trust the direct pairwise call" reference before any added complexity is justified.
Research Program
Fix one patient and one finding, and let the patient have $K$ exams. Let $s_k$ denote the unknown severity of the finding at exam $k$. A pairwise model applied to exams $i < j$ outputs a soft change score $r_{ij} \in [-1, 1]$: the expected value of its improving/stable/worsening softmax, coded as −1, 0, +1. Its ideal value is a monotone function of $s_j - s_i$. With a scale $\omega$ chosen small enough that the full plausible severity range stays well under one turn of the circle, we form a Hermitian matrix. Writing $\iota$ for the imaginary unit and $\bar{\cdot}$ for complex conjugation:
$$H_{ij} = \frac{w_{ij}}{\sqrt{d_i d_j}} \, e^{\iota \omega r_{ij}}, \qquad H_{ji} = \bar{H}_{ij}$$
Here $w_{ij}$ is a pair weight that decays with the time gap between the two exams, and $d_i$ is the weighted degree of exam $i$. The phases of the leading eigenvector of $H$, divided by $\omega$, give the spectral estimate $s^{(0)}$, determined only up to a shared offset.
We then minimize a truncated robust loss starting from $s^{(0)}$. Let $\rho$ be a Tukey-type loss that stops penalizing residuals beyond a cutoff, and let $g$ be a monotone link fit on validation data that maps a severity difference to an expected change score:
$$\min_{s} \sum_{i<j} w_{ij} \, \rho\big(r_{ij} - g(s_j - s_i)\big)$$
We alternate between updating $s$ and refitting $g$. At readout, the change between any two exams is classified by thresholding $s_j - s_i$, with thresholds tuned on validation pairs.
We run this per finding for the five MS-CXR-T conditions. We also run it per anatomical region using the Chest ImaGenome comparison relations, which label whether a region improved, worsened, or did not change between consecutive exams. Running per region tests whether finer-grained synchronization helps.
The pairwise device is a temporal vision-language encoder fine-tuned for three-way progression. We use BioViL-T, whose weights are public, as the main device. If the CoCa-CXR or TempA-VLP releases can be obtained, we repeat with them to show the gain does not depend on one backbone.
Reports nearly always compare a study only to its most recent prior, so ground truth exists mostly for adjacent pairs. The pairwise model, however, can be run on any pair. That asymmetry is the opportunity: labels on the chain, measurements on the whole graph.
The program also studies when synchronization helps. We measure how the gain scales with the number of exams $K$ and with the triangle-inconsistency rate of the raw pairwise outputs. In a controlled setting, we inject synthetic label flips into the pairwise outputs to trace recovery curves against outlier fraction. This is the empirical analogue of the outlier-tolerance results in the phase-retrieval papers.
A secondary use is training. Synchronized trajectories give dense pseudo-labels on non-adjacent pairs, which we can use to fine-tune the pairwise model on long-gap comparisons. We treat this as a follow-up, conditional on the main claim holding.
Proposed Experiments
Data split. All work uses MIMIC-CXR patient histories. Training patients are kept disjoint from every patient appearing in MS-CXR-T. For each MS-CXR-T test pair, we retrieve the patient's full MIMIC-CXR history. We run the pairwise model on all pairs within a fixed time window, synchronize, and read off the change for the benchmark pair.
Primary metric. The primary metric is macro accuracy over the three classes, averaged across the five findings, which is the MS-CXR-T protocol. The direct pairwise prediction on the same pair from the same model is the matched comparison.
Second evaluation. On Chest ImaGenome region-level comparisons, we hide one adjacent pair's label. We then predict that pair once from all other pairs in the history, excluding the direct measurement, and once including it. This separates "the history alone recovers the answer" from "the history corrects the direct answer."
Secondary metrics. These are triangle-inconsistency rate before and after synchronization, expected calibration error of the change readout, and accuracy stratified by $K$ and by projection mismatch (AP vs PA) within the pair.
Baselines
Do-nothing reference. This is the pairwise model's direct prediction on the queried pair, with the rest of the history ignored. It is exactly what BioViL-T, CoCa-CXR, and TempA-VLP do today, and every other method is measured as a difference from it on identical pairs. We also report the trivial "always stable" classifier, to expose class imbalance.
Strongest existing number. The strongest reported result on this benchmark is CoCa-CXR, at 65.0% average MS-CXR-T progression accuracy, 4.8 points above BioViL-T (CoCa-CXR). TempA-VLP reports 61.6% against BioViL-T's 56.3% in a 10% few-shot setting (TempA-VLP). Because our method wraps a pairwise model rather than replacing it, the fair test is device-matched. The same backbone, fine-tuned with the same data and the same budget, is compared with and without synchronization. Beating the baseline means a statistically reliable gain over its own direct prediction, by paired bootstrap over patients. Separately, we report whether synchronization on top of our best device exceeds 65.0%. That comparison is informative but not decisive, since training recipes differ.
Least-squares synchronization. The strongest simple competitor solves $\min_s \sum w_{ij}(r_{ij} - (s_j - s_i))^2$ on the same graph, with the same weights, the same readout thresholds, and the same validation tuning. A chain-composition baseline sums the adjacent-pair calls. It isolates whether non-adjacent measurements add anything.
Ablations isolating the key ingredient. First, the robust refinement is started from least squares and from random starts instead of from the eigenvector. Second, the eigenvector estimate is used alone with no refinement. Third, the degree normalization is removed. Spectral initialization earns its place only if the full method beats both least squares and the least-squares-started refinement. The advantage should widen as the injected outlier rate increases.
Learned multi-exam model. The learned alternative is a small transformer over the $K$ exam embeddings from the same frozen backbone, trained on the adjacent-pair labels with the same compute and tuning budget.
Why not just…?
Why not just fit least squares on the pairwise scores? This is the most serious threat. At low outlier rates, least squares and our method should give nearly the same answer, since both use the whole graph. The case for the spectral-plus-robust estimator rests on the phase-retrieval evidence (Truncated Wirtinger Flow; Median Truncated Wirtinger Flow) that squared-loss estimators are hijacked by a few extreme measurements. Pairs with a projection change plausibly produce such measurements. The outlier-injection curves and the AP/PA-stratified accuracy settle this directly. If least squares ties our method everywhere, we will report the contribution as "multi-pair aggregation" and drop the spectral claim.
Why not just train a sequence model over the whole history? A transformer over all $K$ exams could in principle learn consistency. Two facts argue against it here. Supervision exists almost only on adjacent pairs, so such a model sees little pressure toward global consistency. And CRONOS documents learned sequence models losing to trivial carryover baselines on irregular clinical timelines. Because our method has no new trained parameters beyond a link function and thresholds, it should hold up better on patients with few labeled pairs. The matched-compute transformer baseline tests this, and we will report honestly if it wins on long histories.
Why not just use a stronger pairwise model? Better pairwise models are welcome, and they are inputs to our method rather than rivals. If gains from synchronization vanish as the device improves from BioViL-T to CoCa-CXR-class models, that shows the remaining errors are not inconsistent across pairs. That would itself be a useful finding, and the multi-backbone experiment exists to detect it.
Decisive Checks
Three outcomes would falsify the proposal, and we commit to reporting them. First, with only two exams the method reduces to the direct prediction and must show zero gain. A gain there would signal a bug or leakage. Second, gains must grow with $K$. Third, on the held-out adjacent-pair task, history-only synchronization must recover the hidden label well above the "always stable" rate.
If all three hold, and the eigenvector-started refinement beats both least squares and least-squares-started refinement under realistic outlier rates, then spectral synchronization is a cheap, model-agnostic way past the two-exam limit of current longitudinal clinical models.
References
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
- CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding
- TempA-VLP: Temporal-Aware Vision-Language Pretraining for Longitudinal Exploration in Chest X-Ray Image
- DeltaNet: Conditional Medical Report Generation for COVID-19 Diagnosis
- Normalized Spectral Map Synchronization
- Dual Quaternion SE(3) Synchronization with Recovery Guarantees
- Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems
- Provable Non-convex Phase Retrieval with Outliers: Median TruncatedWirtinger Flow
- CRONOS: Continuous time reconstruction for 4D medical longitudinal series