A 2D ultrasound scan sees a thin slice of a heart that moves in three dimensions. When muscle slides through that slice, the grainy texture of the image (speckle) changes completely. The cause is that different tissue has entered the beam, not that the edge moved. Current boundary-sharpening segmentation networks cannot tell this apart from a genuinely faint edge. They treat both as "ambiguous boundary" and snap the contour to whatever transition the current frame shows. We propose a cheap per-pixel signal that separates the two cases, computed at frame rate. Where speckle has decorrelated at fine scale but the coarse tissue envelope still matches the previous frame, the change was probably caused by through-plane motion. There, the model should trust its temporally propagated shape rather than the current frame's edge. Where both scales track well but contrast is low, the edge is real and boundary refinement should act.
In short, the proposal is a scale-ratio "decorrelation map" that decides, pixel by pixel, whether a real-time echo segmenter sharpens the current edge or carries the previous contour forward. It also decides what the segmenter writes into its temporal memory. The aim is fewer flickering, physically implausible contours and more stable ejection-fraction estimates, within a sub-33 ms per-frame budget.
Core Claim
Our hypothesis is that through-plane motion and in-plane low contrast leave different multi-scale signatures after in-plane motion compensation. We then claim that gating boundary refinement and memory updates on this signature beats gating on the segmenter's own uncertainty. Let $r^{s}_t(x)$ be the smallest normalized patch difference between frame $t$ and frame $t-1$ at pixel $x$ and spatial scale $s$, minimized over in-plane displacements in a small search window. We define the decorrelation score
$$D_t(x) = \frac{r^{\text{fine}}_t(x)}{r^{\text{coarse}}_t(x) + \epsilon},$$
where "fine" is speckle-scale and "coarse" is tissue-envelope scale, and $\epsilon$ is a small constant. A high $D_t$ means texture could not be matched by any in-plane shift even though the large-scale structure could. Ledesma-Carbayo and colleagues identified exactly this as the signature of scatterers leaving the beam (Spatio-temporal nonrigid registration for ultrasound cardiac motion estimation).
A genuinely faint but in-plane edge keeps both residuals low because its speckle tracks, even though its intensity gradient is weak. The segmentation entropy of a network cannot make this distinction: both cases look equally uncertain to it. That is the testable difference.
Where This Sits
The research file shows a clean split.
On one side are methods that model through-plane motion properly but run offline. NeuralCMF fits a continuous 3D motion field per patient and needs about 8.5 minutes on a high-end GPU (Continuous 3D Myocardial Motion Tracking via Echocardiography). MulViMotion and its mesh-based successor recover 3D motion by fusing orthogonal cardiac MRI views, at seconds per subject (MulViMotion, Mesh-based 3D Motion Tracking in Cardiac MRI). These motivate the problem. They also constrain our design: a single handheld view has no second plane to fuse, so we do not try to reconstruct 3D motion. We only detect where the 2D evidence is untrustworthy.
On the other side are fast 2D segmenters with explicit boundary machinery. One adapts the Hiera foundation model at about 77 FPS (Adapting Vision Foundation Models for Real-time Ultrasound Image Segmentation). Another uses reverse attention, which erases confident foreground so later stages focus on ambiguous rims (Parallel Res2Net-based Network with ReverseAttention). MISNet adds axial boundary attention (MISNet), and LSSF-Net uses lightweight focal-modulation context (LSSF-Net). These supply the boundary module we gate. All of them treat every fuzzy rim as a planar contrast problem.
The closest existing work is MemSAM (MemSAM). It filters features with the model's own predicted masks before writing them into a space-time memory. Its authors report that it drifts when early frames are degraded, and that it is too heavy for real-time use. Our gate differs in where its signal comes from. It is computed from raw image matching, independently of the segmenter, so it does not inherit the segmenter's mistakes. It also separates through-plane loss from planar ambiguity, which mask-confidence filtering does not attempt. RecycleNet shows that iterative refinement helps ambiguous boundaries but multiplies latency (RecycleNet). For this reason we restrict ourselves to a single feedforward pass.
Research Program
The system has a 2D encoder-decoder segmenter, built on the Hiera adapter backbone and also on a MobileNet-class backbone for edge hardware. Its decoder ends in a reverse-attention boundary refinement head.
A small temporal state carries two things: the previous frame's contour logits, and a compact memory in the style of DMNet's local aggregation (Efficient Global-Local Memory for Real-Time Instrument Segmentation).
The decorrelation branch runs block matching on the image in two passes. One pass works at quarter resolution with small patches for the fine scale. The other works at eighth resolution with larger patches for the coarse scale. Its search window is sized to the expected per-frame displacement, and it produces $D_t$ plus the coarse displacement field. The previous contour is warped by that coarse field. A learned gate $g_t(x) = \sigma(a \log D_t(x) + b)$, with scalar parameters $a$ and $b$, then forms the boundary logit:
$$\ell_t(x) = \big(1 - g_t(x)\big)\,\ell^{\text{refine}}_t(x) + g_t(x)\,\ell^{\text{warp}}_{t-1}(x).$$
Here $\ell^{\text{refine}}_t$ is the current frame's refined logit and $\ell^{\text{warp}}_{t-1}$ is the warped previous logit. The same gate down-weights memory writes, so frames with heavy through-plane change do not contaminate the stored shape. This addresses MemSAM's drift mode.
The decorrelation map itself is analytic, not learned. We can therefore validate it before any segmentation training. The STRAUS synthetic sequences carry 3D ground-truth motion vectors. We slice them into 2D apical-like planes and compute the true through-plane velocity at each pixel. Training of the segmenter uses CAMUS and EchoNet-Dynamic with standard mask losses, plus a temporal-consistency term on warped predictions.
Baselines
The naive reference is the Hiera-adapter segmenter run independently on every frame, with its reverse-attention head and no temporal state. The adapter paper reports 92.01% Dice on CAMUS at about 77 FPS with TensorRT on an L40s (Adapting Vision Foundation Models). We retrain it on our splits so the comparison uses identical data.
The strongest accuracy competitor is MemSAM. It reports 93.31% mDice on CAMUS-Semi, 92.78% mDice on EchoNet-Dynamic, and an LVEF Pearson correlation of 78.92% (MemSAM). We run the released configuration with its single-point prompt on the same splits, and we report its latency on the same GPU. Beating it means matching its Dice within one point while exceeding its LVEF correlation and temporal stability, at more than 30 FPS where MemSAM cannot run.
The strongest real-time temporal competitor is a DMNet-style local-plus-global memory placed on our backbone (DMNet). DMNet reports 38 FPS on EndoVis17. No reported result was found on Lacuna for DMNet on echocardiography.
Three ablations isolate the key ingredient. The first gates by predictive entropy of the segmenter in place of $D_t$. The second gates by the fine-scale residual alone, with no coarse normalization. The third removes the gate but keeps the warped prior as an extra input channel.
Every variant shares backbone, input resolution, training epochs, and an equal hyperparameter search budget of a fixed number of trials. Every variant is timed end-to-end with the same TensorRT export.
Why not just…?
Why not just smooth the frame-by-frame masks over time? A temporal median or exponential moving average reduces flicker. It does so uniformly, which blurs the real fast motion of the mitral annulus and lags at end-systole. It cannot tell a transient through-plane dropout from a true shape change. We include this smoothing as a baseline and score it by ejection-fraction error, as well as by jitter, because over-smoothing shows up as underestimated volume swings. If it matches us on both, the gate is unnecessary and we will say so.
Why not just gate on the network's own uncertainty? This is the real threat. Entropy is free and already used implicitly by MemSAM's mask-based memory filtering and by DMNet's uncertainty-driven memory updates. Our argument is that a faint in-plane edge and a through-plane dropout are equally uncertain to the segmenter, but only the second should defer to the past. We test this directly. On STRAUS slices, we compare how well $D_t$ and entropy each detect pixels with high true through-plane velocity, measured by area under the detection curve. On CAMUS and EchoNet-Dynamic, the entropy-gated ablation is trained identically. If entropy detects through-plane pixels as well as $D_t$, or the entropy-gated model matches ours, the core claim fails.
Why not just use a 2D+time convolutional network or a full space-time memory transformer? Dense 3D convolutions are far slower than interactive frame rates. In MulViMotion's comparison, a 3D-UNet was also less accurate (Dice 0.73 against 0.82, MulViMotion). Space-time attention is what makes MemSAM accurate but slow. Both learn an implicit temporal prior and offer no explicit control over when to trust the current frame. We still report MemSAM's accuracy, so a reader can see what our speed costs.
Proposed Experiments
The first decisive check validates the signal before any training. On STRAUS-derived 2D planes, we measure whether $D_t$ ranks pixels by true through-plane velocity better than segmentation entropy and better than the fine residual alone.
A controlled corruption test on CAMUS frames follows. In one condition we lower contrast and blur a myocardial segment while keeping its speckle consistent across frames, which simulates a faint in-plane edge; $D_t$ should stay low. In the other we replace that segment's speckle with independent speckle, which simulates through-plane exit; $D_t$ should fire. If $D_t$ responds to the contrast change, the decomposition is not working.
The main evaluation trains all baselines and ablations on CAMUS and EchoNet-Dynamic. We report Dice and Hausdorff distance on annotated frames, frame-to-frame area jitter between annotated frames, and LVEF Pearson correlation and absolute error. Results are stratified by the mean $D_t$ inside the myocardium, so that gains, if any, should concentrate in high-decorrelation frames such as basal segments during longitudinal shortening.
Latency is measured end-to-end per frame on an L40s with TensorRT, and on an edge-class embedded GPU. The file notes that published latencies come almost entirely from desktop GPUs. The pass criterion is under 33 ms per frame on the edge device for the MobileNet variant.
If held-out handheld point-of-care scans are available, we test generalization there. Fine-tuned SAM fell from 0.902 on standard transthoracic echo to 0.857 Dice on handheld scans (Foundation versus Domain-Specific Models), and probe motion makes through-plane change more common on such devices. If no such data is available, we will report this as untested rather than claim it.
References
- Spatio-temporal nonrigid registration for ultrasound cardiac motion estimation
- Continuous 3D Myocardial Motion Tracking via Echocardiography
- MulViMotion: Shape-Aware 3D Myocardial Motion Tracking From Multi-View Cardiac MRI
- Mesh-Based 3D Motion Tracking in Cardiac MRI Using Deep Learning
- Adapting Vision Foundation Models for Real-Time Ultrasound Image Segmentation
- Parallel Res2Net-based Network with ReverseAttention for Polyp Segmentation
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- LSSF-Net: Lightweight segmentation with self-awareness, spatial attention, and focal modulation
- MemSAM: Taming Segment Anything Model for Echocardiography Video Segmentation
- Abstract: RecycleNet - Latent Feature Recycling Leads to Iterative Decision Refinement
- Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video
- Foundation versus Domain-Specific Models for Left Ventricular Segmentation on Cardiac Ultrasound