Training a network to outline the heart's left ventricle in ultrasound video normally takes thousands of annotated clips. Many real cohorts, such as a rare cardiomyopathy or a single-site trial, have a few dozen. This proposal builds synthetic training videos from such a cohort by recombining three things that belong to different patients: the heart's shape over time from patient A, the ultrasound texture (speckle, blood-pool darkness, myocardial brightness) from patient B, and a timing correspondence between them so that B's texture at a given point in the cardiac cycle lands on A's shape at the same point. The texture is produced by optimization-based synthesis, a training-free method that starts from noise and adjusts pixels until summary statistics of a fixed filter bank match those of a reference. Here those statistics are computed separately inside each anatomical region and matched frame by frame along a dynamic time warping alignment. Dynamic time warping, or DTW, is a dynamic-programming method that pairs up frames of two sequences that move at different speeds. Every synthetic clip inherits patient A's masks exactly, so it comes labeled by construction. With $n$ patients the method can produce on the order of $n^2$ shape–texture pairings without training any generator.
The motivation is that the two obvious ways to get more data both fail at this scale. Feedforward video generators need hundreds of clips just to learn motion. Single-exemplar texture synthesis needs only one, but as used today it matches statistics over the whole frame and assumes motion is stationary, and the heart violates both assumptions. The claim to test is narrow: in the $n \le 45$ regime, label-consistent recombination of per-region texture, aligned by cardiac phase, improves left-ventricle segmentation and ejection-fraction estimation beyond what strong conventional augmentation and few-shot fine-tuning of a foundation segmenter achieve.
Where this sits in prior work
Optimization-based synthesis supplies the core method. Gatys et al. introduced texture synthesis by matching Gram matrices, which are channel co-activation statistics pooled over all positions, of a frozen network. What does it take to generate natural textures? showed that a single layer of untrained random filters at eight scales works as well. We reuse that random multiscale bank because it carries no natural-image bias, which matters for speckle. Two-Stream Convolutional Networks for Dynamic Texture Synthesis extended synthesis to video by adding a motion-energy stream optimized with a quasi-Newton method (L-BFGS). It pools over the entire volume and reports failure on non-stationary motion. A heartbeat is non-stationary: rapid ejection, slow diastasis, and an atrial kick happen at different speeds, and cycle lengths vary. We differ from it in two ways. Statistics are pooled within anatomical regions rather than over the whole frame, and the target statistics change frame by frame according to a phase alignment rather than being one stationary summary.
Region-of-interest work supplies the pooling idea and the motivation. Mask-weighted global average pooling for CT triage weights feature aggregation by a lung mask and reports gains over unweighted pooling. The Role of Pleura and Adipose in Lung Ultrasound AI shows that unmasked ultrasound classifiers latch onto subcutaneous tissue, and masking raises AUC from 0.6553 to 0.7742. Unveiling COVID-19 from Chest X-ray with deep learning shows that small-data models cluster by source rather than disease, and that shallower networks generalize better. These papers use masks to constrain discriminative models. We put the mask inside a synthesis objective. That creates a problem none of them face: convolutional filters straddling a region boundary contaminate the region's statistics.
The small-sample cardiac work constrains the design. Dong et al. segment the left ventricle from only 45 3D sequences by combining a shape prior with a deformable snake, but they treat end-diastole and end-systole as static frames. UniLVSeg learns cardiac dynamics by masked-frame pretraining on 10,030 videos, which is precisely the resource a small cohort lacks. Ultrasound Image-to-Video Synthesis via Latent Dynamic Diffusion Models shows that synthetic ultrasound video can help downstream classifiers, but it needs a trained video autoencoder and reports implausible motion from its higher-diversity competitor. Our method sidesteps learned motion entirely, because motion comes from real mask sequences.
For alignment, D3TW makes DTW differentiable and adds a margin loss against degenerate alignments, but only for 1D action transcripts. shapeDTW aligns local shape descriptors rather than raw values, which reduces the noise-chasing documented for raw biosignals in From Continuous sEMG Signals to Discrete Muscle State Tokens. We use shapeDTW-style descriptors on a 1D physiological signal derived from the masks. To our knowledge, nothing in this literature uses DTW to schedule the targets of pixel-level synthesis.
Core mechanism
For each training patient we take the per-frame masks of left-ventricular cavity, myocardium, left atrium, and background. From these we compute the cavity-area curve and its derivative, and describe each frame by a short window of that derivative, in the manner of shapeDTW. DTW with a Sakoe-Chiba band, a cap on how far the alignment may stray from the diagonal, aligns patient A's curve to patient B's. The result is a path $\pi$ mapping each target frame $t$ of A to a possibly fractional frame $\pi(t)$ of B. Fractional positions are realized by interpolating B's statistics. This avoids the zero-motion artifact that a repeated frame index would otherwise create.
Let $F^{l}(X_t)$ denote random-filter responses at scale $l$ for the synthesized frame $X_t$. Let $m^{r}_{t}$ denote region $r$'s mask in A, eroded by the filter's receptive-field radius so that interior statistics never see the boundary. The region Gram matrix is $G^{l,r}_{ij}(X_t)=\sum_k m^r_{t,k}F^l_{ik}F^l_{jk}/\sum_k m^r_{t,k}$. Here $k$ indexes pixels and $i,j$ index channels. The same quantity is computed on B's frame $Y_{\pi(t)}$ under B's own eroded mask. We then minimize
$$\mathcal{L}(X)=\sum_t\sum_{r,l}\big\|G^{l,r}(X_t)-G^{l,r}(Y_{\pi(t)})\big\|^2+\lambda\sum_t\sum_{r}\big\|G^{r}(X_{t+1}-X_t)-G^{r}(Y_{\pi(t+1)}-Y_{\pi(t)})\big\|^2,$$
over the pixels of the whole clip $X$, starting from noise and optimizing with L-BFGS. The second term matches statistics of frame-to-frame differences. It is a region-wise stand-in for Tesfaldet et al.'s motion stream, and it captures how quickly speckle decorrelates as tissue moves. Its weight is $\lambda$. Pixels inside the boundary band, which belong to no eroded region, are covered by an additional band-region term. That term matches B's band statistics, which carry the edge contrast a segmenter must learn. Shape is never optimized: it is fixed by A's masks through which pixels each region term acts on, so labels are exact by construction. Diversity comes from the A×B pairing and from the noise initialization. That directly addresses the near-reconstruction problem that Ustyuzhaninov et al. note for single-exemplar synthesis.
Research Program
The first stage builds and inspects the synthesizer on CAMUS, which the research file describes as having dense per-frame multi-structure annotations. If only end-diastole and end-systole turn out to be labeled for some views, masks will be propagated between them and that propagation step reported. We will measure boundary leakage with two diagnostics. One is the gap between region statistics computed on synthesized frames and the exemplar targets. The other is the visibility of halos at mask edges, which we expect to appear if erosion is turned off. We will also measure how the synthesizer's wall-clock cost scales with clip length and resolution. Optimization synthesis is known to be slow, and the method is only useful if a few hundred clips can be produced within a fixed GPU budget that we will report.
The second stage asks the downstream question. A small 2D+time segmenter is trained on $n\in\{10,20,45\}$ real patients plus synthetic clips. We use a ResNet-18-scale encoder, following the evidence that shallow models generalize better in this regime. We then evaluate on held-out CAMUS patients and, for distribution shift, on the EchoNet-Dynamic test split, using Dice on end-diastole and end-systole frames and ejection-fraction error. The cohort size of 45 deliberately matches the CETUS scale used by Dong et al. As a secondary check, the synthesizer will be applied slice-wise to CETUS volumes and evaluated with mean surface distance, which Dong et al. report as 2.20 mm at end-diastole and 2.56 mm at end-systole on that dataset.
Baselines
The naive reference is the same segmenter trained only on the $n$ real patients with flips, small affine transforms, and intensity jitter. Everything else must beat it, and its gap to full-data performance sets the size of the opportunity. Full-data ceilings come from the file. SimLVSeg in UniLVSeg reports 93.32% Dice on the EchoNet-Dynamic test set. Foundation versus Domain-Specific Models reports Dice 0.911 for box-prompted fine-tuned SAM on the EchoNet internal test set, and 0.891 when transferring to CAMUS two-chamber views after training on four-chamber views. All of these numbers come from training on thousands of videos. No reported result found on Lacuna for any of these methods at $n\le45$ patients, so we will rerun them ourselves in the small regime.
The strongest existing competitor is box-prompted fine-tuned SAM, the Segment Anything Model adapted as in Chao et al., fine-tuned on the same $n$ patients. It is the strongest because its large pretrained prior is the main alternative source of knowledge when labels are scarce, and in the file it degrades least under scanner shift, holding Dice 0.857 on handheld point-of-care ultrasound where EchoNet falls to 0.667. It will get the same patients, the same tuning budget in GPU-hours, and the same oracle-free box protocol at test time, with boxes derived from a detector trained on the same $n$. As a separate arm, we also report our synthetic data added to SAM fine-tuning, since the two are complementary.
The strongest augmentation competitor is a carefully tuned conventional pipeline: elastic spatial deformation, speckle-like multiplicative noise, gamma and contrast jitter, and random nonlinear temporal resampling of real clips. Its hyperparameters get the same search budget as our $\lambda$, erosion radius, and synthetic-to-real ratio. A cheap texture-recombination baseline isolates whether the optimization machinery matters. It fills A's regions with B's texture using region-wise histogram matching and random-phase noise that matches each region's power spectrum. That reproduces second-order statistics without iterative synthesis.
Four ablations isolate the key ingredients. The first is whole-frame Gram matching with no masks, in the style of Tesfaldet et al. The second is region masks without erosion, which tests the boundary-artifact concern. The third replaces DTW with piecewise-linear alignment anchored at end-diastole and end-systole. The fourth drops the difference term ($\lambda=0$). Beating a baseline means a gain of at least two Dice points or a significant reduction in ejection-fraction error, measured by paired bootstrap over test patients across five seeds of patient subsampling, with the effect holding on both CAMUS and EchoNet-Dynamic.
Why not just…?
Why not just train a video diffusion model on the small cohort? Latent diffusion for ultrasound video, as in Chen et al., depends on an autoencoder and a motion model learned from hundreds of clips, and that work itself reports that a higher-diversity generator produced implausible lesion motion that hurt downstream accuracy. A generator trained on 20 patients has nowhere to learn cardiac kinematics. Our method takes motion directly from real masks. We will still include a diffusion model fine-tuned on the $n$ patients from public pretrained video weights, if the compute budget allows. If it matches us, the claim narrows to cost and label consistency.
Why not just fine-tune SAM with a few labeled patients? This is the most serious threat, because fine-tuned SAM transfers unusually well in Chao et al. Even so, it processes frames independently, needs box prompts, and has never been shown to work at $n\le45$. The head-to-head comparison and the SAM-plus-synthesis arm settle this directly. If SAM alone wins and our data adds nothing on top, the proposal's practical value is refuted.
Why not just use strong conventional augmentation, including random temporal warping of real clips? Conventional augmentation perturbs one patient's appearance and timing. It never pairs one patient's anatomy with another's speckle and blood-pool contrast, and that pairing is the kind of variation that differs across scanners and patients. Whether that variation matters is exactly what the tuned-augmentation baseline and the histogram/random-phase recombination baseline test. If cheap recombination matches the optimized version, then the synthesis machinery is unnecessary, and we will report that.
Why not just align phases linearly between end-diastole and end-systole? For the systolic half of the cycle, two anchors may well be enough. DTW should matter in diastole, where rapid filling, diastasis, and the atrial kick move at very different speeds, and in patients with irregular cycles. The anchored-linear ablation will be scored on diastolic-frame Dice specifically. If it ties DTW there, we adopt the simpler alignment and drop DTW.
Decisive checks
The proposal stands or falls on three outcomes at $n=20$. Phase-aligned masked synthesis must beat tuned conventional augmentation on both CAMUS and EchoNet-Dynamic. It must beat the random-phase recombination baseline, otherwise iterative optimization is not earning its cost. And whole-frame unmasked synthesis must do measurably worse, confirming that region-restricted statistics, not just extra data, drive the gain. A further diagnostic tests the mechanism itself. If recombination helps through cross-patient texture, then gains should grow when A and B come from different acquisition conditions, and shrink when textures are drawn only from the same patient as the shape.
References
- Texture Synthesis Using Convolutional Neural Networks
- What does it take to generate natural textures?
- Two-Stream Convolutional Networks for Dynamic Texture Synthesis
- Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling
- The Role of Pleura and Adipose in Lung Ultrasound AI
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- A Left Ventricular Segmentation Method on 3D Echocardiography Using Deep Learning and Snake
- UniLVSeg: Unified Left Ventricular Segmentation with Sparsely Annotated Echocardiogram Videos through Self-Supervised Temporal Masking and Weakly Supervised Training
- Ultrasound Image-to-Video Synthesis via Latent Dynamic Diffusion Models
- D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
- shapeDTW: shape Dynamic Time Warping
- From Continuous sEMG Signals to Discrete Muscle State Tokens: A Robust and Interpretable Representation Framework
- Foundation versus Domain-Specific Models for Left Ventricular Segmentation on Cardiac Ultrasound