Motion You Can Trust Across Scanners: Estimating Cardiac Flow on Sketch-Like Boundary Maps

Video models of the beating heart work better when they line up each frame's internal features with its neighbours before combining them. They do this by estimating how every point moved, a quantity called optical flow, and using it to pull features from one frame into the coordinates of another. The trouble is that flow is usually estimated from raw image brightness. When a model trained on one MRI vendor is used on another, the contrast, noise and reconstruction artifacts change, so the brightness-based motion estimate degrades precisely when alignment is needed most. This proposal estimates motion on a different input: a thin, sketch-like map of anatomical boundaries that a small network draws for each frame. The idea borrows from sketch-to-photo retrieval, where removing texture and keeping only geometry is a known way to close the gap between visually different sources of images.

The claim is narrow and testable. In cine cardiac MRI, which is a short movie of the heart over one beat, motion-compensated feature alignment should improve segmentation on an unseen scanner vendor only if the motion is estimated in a representation that is stable across vendors. Concretely, flow computed between learned boundary maps should transfer to an unseen vendor much better than flow computed from intensities. That advantage should be small on the vendors seen in training and large on the unseen one. If the gap does not open on the unseen vendor, the idea is wrong.

What exists and what is missing

The temporal-alignment literature supplies the mechanism. Flow-Guided Feature Aggregation for Video Object Detection warps feature maps from clean neighbouring frames onto degraded ones. Semantic Video CNNs Through Representation Warping shows that warping intermediate activations, rather than pixels, keeps boundaries sharp. Semantic Video Segmentation by Gated Recurrent Flow Propagation keeps a recurrent memory aligned to the moving scene with flow, which prevents spatial drift. In medicine, FW-Net warps semantic features in fluoroscopy using a lightweight FlowNet. It reaches Dice 0.821 against 0.677 for U-Net, but only on a single phantom setup, and its flow is computed on raw intensities. We reuse this warping machinery unchanged. What we change is the input from which motion is estimated.

The cardiac generalization work motivates the gap. cineCMR-SAM adds divided temporal-spatial attention to the Segment Anything Model (SAM), a large promptable image segmenter. It reports zero-shot Dice 0.835 on the multi-vendor M&Ms dataset against 0.810 for U-transformer. Its temporal attention compares the same pixel coordinate across frames, so it ignores how far the myocardium moves between relaxation and contraction. MediViSTA weights neighbouring echo frames with a fixed Gaussian window at identical coordinates and injects wavelet high-frequency features. It reports a 2.15% Dice gain over the second-best model on unseen data. Both methods use edge-like signals or temporal context, but neither aligns along motion. RL4Seg3D enforces temporal smoothness through a reinforcement-learning penalty across 357 echo centres. Its authors note that very rapid cardiac motion remains unsolved, and the heuristic smoothness term is no substitute for motion-tracked alignment. The M&Ms challenge documents the drop on unseen vendors that we target.

The cross-modal literature supplies the representational idea. Sketch-based retrieval closes the sketch–photo gap by reducing photos to structure. Semi-Heterogeneous Three-Way Joint Embedding Network does this with edge maps used as a stepping stone, and Contour detection network for zero-shot SBIR does it with a dedicated contour detector. That literature also constrains our design. Automatically extracted edges are cluttered with background detail, while human sketches keep only meaningful outlines. Asymmetric Mutual Alignment attributes persistent gaps to this asymmetry in information density. For that reason we do not use generic edges. We train the boundary drawer to output only anatomical contours: the endocardium, epicardium and right-ventricle borders, supervised by the edges of ground-truth masks. In that sense it behaves more like a clinician's sketch than an edge detector.

Two medical papers are the nearest structural cousins. GauSAM injects contourlet features into SAM and transfers well across polyp datasets. Learning Active Contour Models adds a boundary-length loss on ACDC. Both are static and work frame by frame. We avoid the generative branch of cross-modal work, such as Generative Domain-Migration Hashing, which translates sketches into synthetic photos. Its known hallucination risk would leak invented structure into the motion estimate.

Core Claim

Flow-guided alignment and structural abstraction have each been tried, alignment for temporal stability and abstraction for cross-domain matching. They have not been combined in the specific way that matters here, with the abstraction placed inside the motion estimator. The prediction is that intensity-based flow is the weak link under vendor shift. Moving motion estimation into a boundary space should therefore make warped-feature fusion help more on unseen vendors than on seen ones. We test this through the quality of the motion field itself, not only through final segmentation Dice.

Research Program

The system has three parts on a shared backbone. The backbone is the cineCMR-SAM design, a SAM image encoder with temporal adapters, so that comparisons isolate the alignment step.

The first part is a light boundary head. It takes each frame and outputs a soft map of anatomical contours, trained with a boundary loss against the outlines of annotated masks. It is also trained with an intensity-perturbation consistency term: the map should not change under random contrast curves, noise and blur.

The second part is a small flow network. It takes boundary maps from two frames and predicts a displacement field $\mathbf{u}$. Its training signal has three components. The first is a reconstruction term in which the warped boundary map of frame $t-1$ should match that of frame $t$. The second is a smoothness term that fills in motion inside the sparse rings. The third is supervised wherever labels exist. Because M&Ms and ACDC annotate end-diastole (ED, the relaxed heart) and end-systole (ES, the contracted heart), we compose flow across the frames between them and require the warped ED mask to match the ES mask.

The third part is the alignment step. Backbone features from neighbouring frames are warped with $\hat F_{t-1}(x) = F_{t-1}(x+\mathbf{u}(x))$, where $F$ are feature maps and $x$ a pixel location. They are then fused through a gated recurrent unit in the style of Nilsson and Sminchisescu. The gate learns to down-weight regions where boundary-map reconstruction error is high. Such regions are a cue for through-plane motion, which bilinear warping cannot represent.

The analysis has two strands. First, we measure flow quality separately on seen and unseen vendors, using ED-to-ES mask propagation Dice. This tests the mechanism directly, rather than inferring it from segmentation scores. Second, we examine where along the cycle the gains occur. The prediction is that they concentrate in mid-systole, when wall velocity is highest.

Baselines

The naive reference is a per-frame 2D segmenter: the same SAM backbone with adapters but no temporal module, trained with standard augmentation. It shows how much any temporal modelling contributes.

The strongest existing method is cineCMR-SAM itself, retrained on our split. Its reported zero-shot M&Ms Dice is 0.835 against 0.810 for U-transformer (cineCMR-SAM). That figure was obtained after fine-tuning on STACOM2011, so we will report both its published protocol and a retrain on our M&Ms vendor split, using the same data, epochs and number of tuning trials as our method. Beating it means a statistically significant Dice gain on the held-out vendors across patient-level bootstrap resamples, with no loss on seen vendors.

MediViSTA's Gaussian temporal window with wavelet edge injection is the second strongest competitor. It already combines edge features and temporal context without motion. We reimplement it on the same backbone and give it a matched tuning budget.

The key ablation keeps everything identical except the flow input, which takes one of four forms. The first is raw intensities. The second is intensities with the same heavy contrast augmentation the boundary head receives. The third is wavelet high-frequency bands. The fourth is our learned boundary maps. A last ablation replaces the anatomy-only boundary head with a generic edge detector, which tests the sketch-versus-edge-map argument.

Why not just…?

Why not just augment intensities heavily so ordinary flow becomes vendor-robust? This is the most serious threat. Strong contrast and noise augmentation may already make an intensity flow network invariant enough. The augmented-intensity ablation addresses it directly, because it receives the identical perturbation set used to train our boundary head. If its propagation Dice on the unseen vendor matches ours, the structural bridge adds nothing and we will report that.

Why not just let attention learn the alignment implicitly? Temporal attention could in principle attend to displaced locations. cineCMR-SAM's divided attention is constrained to fixed coordinates, however, and MediViSTA's window is fixed by design. Unconstrained spatiotemporal attention is the obvious alternative, and we add a full space-time attention variant at matched parameters. If it equals boundary-flow warping on held-out vendors, explicit motion is unnecessary. Our expectation, that explicit motion is still needed, rests on FW-Net's and Zhu et al.'s gains from explicit warping. That evidence does not come from cross-vendor settings, so this is an open empirical question.

Why not just feed edge or wavelet features to the segmenter and skip motion? GauSAM and MediViSTA show that structural features help transfer even without alignment. A version that injects our boundary maps into the segmenter but does not warp tests whether the gain comes from the bridge as an input feature rather than from motion estimated on it. Our claim requires the warped model to beat this version specifically on high-velocity systolic frames.

Why not just use reinforcement-learning shape and temporal rewards as RL4Seg3D does? That approach adapts to the target domain with substantial compute and needs target data. Ours is zero-shot at the unseen vendor. RL4Seg3D reports 94.2% Dice on echo, but no reported result was found on Lacuna for it on cine MRI M&Ms. We therefore treat it as complementary: its anatomical reward could later be added on top of our model.

Proposed Experiments and Decisive Checks

The primary benchmark is M&Ms. We train on Siemens and Philips and test on GE and Canon, the challenge's unseen-vendor setting. ACDC and STACOM2011 serve as secondary transfer targets. The metrics are Dice and Hausdorff distance for the left ventricle, myocardium and right ventricle at ED and ES, and ejection-fraction error. Temporal consistency is measured as agreement between each frame's mask and its flow-warped neighbour, computed with a fixed independent flow so that methods cannot score themselves. Flow quality is measured as ED-to-ES propagation Dice.

The decisive check is an interaction effect. The advantage of boundary-map flow over augmented-intensity flow in propagation Dice should be near zero on seen vendors and clearly positive on unseen vendors, and segmentation gains on unseen vendors should track that flow gap across patients. A second check is a velocity analysis: gains over cineCMR-SAM should be largest in the frames with the most wall motion. A third check is a failure audit of basal slices, where through-plane motion breaks two-dimensional warping. There we expect the gate to fall back toward unwarped features, and we will report the cases where it does not.

If boundary flow fails to beat augmented-intensity flow on held-out vendors, the conclusion is that the brittleness lies in the intensities and is solved by augmentation, not by structural abstraction. That is still a useful and reportable negative result for the field.