When a network segments the heart wall in an ultrasound video, it usually borrows evidence from neighboring frames. It looks a fixed number of frames forward and backward. That works if every heartbeat lasts the same time, but real heart rate speeds up and slows down within a single clip, and in arrhythmias it can change from one beat to the next. A fixed window of "three frames back" then sometimes lands in the same part of the beat and sometimes in a different one. When it lands in a different part, the network averages a contracted wall with a relaxed one and smears a boundary that was already faint. We propose to index temporal convolutions by cardiac phase (how far through the current beat a frame is) instead of by frame count. A small motion model estimates each frame's phase. The convolution then gathers features from frames at fixed phase offsets: a sliver of a beat away for fast wall motion, a quarter-beat away for slower filling, and exactly one beat away to reach the same contractile state in the neighboring heartbeat.
The last branch matters most for faint boundaries. When acoustic dropout or shadowing erases part of the endocardium in one frame, the frame exactly one cycle earlier or later shows the same anatomy in the same state and often has better contrast. A phase-indexed operator can borrow that evidence directly, while a frame-indexed one cannot unless heart rate happens to be constant. The bet is modest and testable. Gains should be negligible on clips with steady rhythm and should grow with within-clip rhythm variability and with boundary contrast loss.
Core Claim
Replacing frame-indexed temporal kernels with phase-indexed, multi-scale temporal kernels in a standard echocardiography segmentation network will improve endocardial delineation on clips with irregular rhythm and on low-contrast frames. It will cost little on regular clips and does not require a memory bank. The novelty is the temporal operator itself, a temporal convolution whose sampling positions come from an estimated instantaneous phase. It is neither a new segmentation architecture nor a new boundary loss.
The idea combines three lines of prior work that have not been joined. First, from non-stationary forecasting, CycleNet (CycleNet) shows that removing an explicit learned cycle lets a tiny model beat large Transformers. Its cycle length is fixed a priori, though, which is exactly what fails under heart rate variability. Frequency Adaptive Normalization (FAN) tracks dominant frequencies per instance but works through a global DFT over the window, so it cannot follow a period that changes within the window. We keep CycleNet's "compare against the same point in the cycle" principle and let the cycle length vary continuously.
Second, from multi-scale temporal convolution, TSception (TSception) sets parallel kernel lengths as fixed ratios of the sampling rate. LGGNet (LGGNet) reweights those branches per instance. We set our branch scales as ratios of the instantaneous heart period rather than the sampling rate, and we reuse LGGNet-style attentive fusion, applied per pixel.
Third, from low-contrast boundary work, MemSAM (MemSAM) documents that memory banks fail when corrupted low-contrast frames enter memory. That motivates an aggregation rule that does not depend on frame features being clean. MISNet (MISNet) supplies a boundary-supervision recipe, which we apply equally to every model so that it is not confounded with our contribution.
Research Program
The phase estimator follows Latent Motion Profiling (LMP). That work trains an autoencoder that splits each frame into a static anatomy code and a two-dimensional motion code, and the motion code traces an ellipse once per beat. LMP uses the extrema of this ellipse to find end-diastole and end-systole. We instead read off the unwrapped angle around the ellipse center as a continuous phase $\phi(t)$, measured in radians and increasing by $2\pi$ per beat. This gives a phase for every frame without labels.
The new operator samples features at fixed phase offsets:
$$ y_t = \sum_{k} w_k \, x\big(\phi^{-1}(\phi(t) + \Delta_k)\big) $$
Here $x(s)$ is the feature map at a possibly fractional frame time $s$, obtained by linear interpolation between neighboring frames. $\phi^{-1}$ maps a phase back to the frame time at which it occurs. $\Delta_k$ are fixed phase offsets, and $w_k$ are learned weights.
We use three branches:
- a fine branch with offsets of about $\pm 2\pi/32$, for fast wall motion near systole;
- a medium branch with offsets of about $\pm 2\pi/8$;
- a cycle branch with offsets of $\pm 2\pi$ and $\pm 4\pi$, which reaches the same phase in adjacent beats.
Coarse branches skip frames, so we low-pass the features along the phase axis before sampling. This follows the aliasing result of Anti-Aliasing Matters. A per-pixel gate fuses the branches. The gate also receives a cheap local contrast estimate, so the network can lean on the cycle branch where the current frame is degraded.
The host network is the SimLVSeg 3D U-Net (SimLVSeg). We keep its spatial convolutions, self-supervised temporal-masking pretraining, and weak supervision through end-diastole and end-systole frames. We swap only the temporal kernels in the bottleneck and the two coarsest decoder stages for the phase-indexed operator. The phase estimator is pretrained, then frozen in the main variant and fine-tuned through the interpolation in a second variant.
Baselines
Naive reference. A per-frame 2D U-Net with no temporal modeling, trained under the same weak supervision. It shows how much any temporal aggregation helps.
Strongest existing method. SimLVSeg, which reports a Dice of 93.32% (95% CI 93.21–93.43%) on the EchoNet-Dynamic test set (SimLVSeg). It is the right competitor because our model is SimLVSeg with one operator changed. We retrain it ourselves with identical pretraining, epochs, augmentation, boundary loss and hyperparameter search budget, and report both our reproduction and the published figure.
Memory-based alternative. MemSAM, reporting 92.78% mDice on EchoNet-Dynamic, 93.31% on CAMUS-Semi, and 78.92% ejection-fraction Pearson correlation (MemSAM). It represents the competing way of aggregating across beats, using similarity-based memory lookup instead of phase indexing. We run its released configuration with its single first-frame prompt, and we also run it with the prompt placed on our model's lowest-contrast frame to probe the initialization fragility its authors note.
Foundation model. Fine-tuned SAM, reported at Dice 0.911 on EchoNet-Dynamic and 0.857 on noisy point-of-care ultrasound (Foundation versus Domain-Specific Models). The point-of-care cohort is private to Mayo Clinic, so we cite that number for context only.
Ablations isolating the key ingredient. Five variants hold everything else fixed:
- the same three-branch bank with frame-indexed offsets equal to the dataset's median beat length (multi-scale, no phase);
- the phase-indexed operator with only the cycle branch;
- the phase-indexed operator without the cycle branch;
- deformable temporal convolution with freely learned offsets and no phase supervision;
- the full model with the phase replaced by a constant-rate phase fitted to the clip's average heart rate.
The constant-rate variant is the critical one. If it matches the full model, within-clip drift is not what drives the gains.
What counts as beating SimLVSeg. Aggregate Dice on EchoNet-Dynamic sits near a ceiling, so a tie there is acceptable. Beating it means a confidence-interval-separated improvement in boundary error (95th-percentile Hausdorff distance and boundary F-score) on the top tercile of rhythm variability and the bottom tercile of boundary contrast, plus no regression on the remaining clips. No reported result found on Lacuna for Hausdorff or boundary F-score with any of these baselines, so we compute both for all of them.
Why not just…?
Why not just detect the beats, resample each clip to a fixed beat length, and run SimLVSeg? This is the most serious alternative. If heart rate is roughly constant within each clip, global resampling plus random-speed augmentation captures nearly all of the benefit, and our method reduces to a complicated version of it. The answer is empirical. We include a "resample-then-SimLVSeg" baseline using the same phase estimator to set a per-clip speed. We also include speed augmentation as a training option for every baseline. Then we stratify results by within-clip phase-rate variability. If our gains vanish once resampling is added, we will report that phase indexing adds nothing beyond per-clip normalization in this data. The claim then narrows to arrhythmic clips specifically.
Why not just use deformable temporal convolutions and let the network learn where to look? Learned offsets can in principle discover phase alignment. But under supervision on less than 1.2% of frames, which is EchoNet-Dynamic's label density, the training signal is too thin to learn cross-beat correspondence reliably. A learned offset also has no reason to land exactly one beat away. The deformable ablation tests this directly, and we will inspect whether its learned offsets correlate with estimated phase.
Why not just rely on MemSAM-style memory attention, which already matches similar frames across beats? Memory lookup matches by feature similarity, and that is exactly what degrades when a frame's boundary drops out (MemSAM). Phase comes from a global two-dimensional motion trajectory pooled over the whole frame, so local dropout should perturb it less. This remains a real threat on clean data, where memory may perform just as well. The synthetic-dropout experiment below is designed to separate the two.
Why not just fine-tune SAM frame by frame? The fine-tuned SAM numbers above (0.911 on EchoNet-Dynamic, 0.857 on point-of-care data) show strong spatial priors. But frame-by-frame prediction has no way to recover a boundary that is absent in the current frame, and MemSAM reports that a frame-wise SAMUS model reaches only 67.55% ejection-fraction correlation versus 78.92% with temporal memory. We include it to quantify the size of the temporal contribution.
Proposed Experiments
The data are EchoNet-Dynamic for training and main testing. CAMUS-Full, with dense per-frame labels, lets us evaluate mid-cycle frames and frame-to-frame consistency. CAMUS-Semi provides comparison with MemSAM's protocol. Metrics are Dice, 95th-percentile Hausdorff distance, boundary F-score, temporal jitter of the predicted boundary on CAMUS-Full, and ejection-fraction correlation and absolute error.
We stratify the test sets along two axes. Rhythm variability is the coefficient of variation of the phase rate within each clip, computed from the frozen estimator. Boundary contrast is the intensity ratio between thin bands just inside and just outside the ground-truth contour. We validate the phase estimator first, checking its end-diastole and end-systole timing against LMP's reported errors of 3.0 and 2.0 frames on EchoNet-Dynamic (LMP). A badly wrong phase would invalidate everything downstream.
The first decisive check is controlled drift. We resample clips non-uniformly in time with a known, smoothly varying rate profile. Labeled frames are kept, and frames are selected, not interpolated, to avoid synthesis artifacts. We then plot boundary error against drift amplitude for our model, the constant-rate ablation, resample-then-SimLVSeg, and MemSAM. The prediction is that only the phase-indexed model stays flat as drift grows.
The second decisive check is controlled dropout. We blank or shadow a wedge of the boundary in labeled frames and measure recovery. The prediction is that the cycle branch recovers the occluded segment, and that MemSAM degrades more when the corrupted frame is its prompt frame.
The third check is the natural-data stratification. The method is only worth adopting if its advantage over the resampling baseline appears in the high-variability, low-contrast stratum of real clips, not just under synthetic warps. If the advantage appears only under synthetic warps, we will say so. We would then recommend the simpler resampling pipeline for routine echocardiography and reserve phase indexing for arrhythmia-heavy cohorts.
References
- CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns
- Frequency Adaptive Normalization For Non-stationary Time Series Forecasting
- TSception: A Deep Learning Framework for Emotion Detection Using EEG
- LGGNet: Learning from Local-Global-Graph Representations for Brain-Computer Interface
- MemSAM: Taming Segment Anything Model for Echocardiography Video Segmentation
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- Latent Motion Profiling for Annotation-free Cardiac Phase Detection in Adult and Fetal Echocardiography Videos
- Anti-Aliasing Matters: A Dynamic Network for Time Series Forecasting
- SimLVSeg: Simplifying Left Ventricular Segmentation in 2D+Time Echocardiograms with Self- and Weakly-Supervised Learning
- Foundation versus Domain-Specific Models for Left Ventricular Segmentation on Cardiac Ultrasound