Video networks for medical ultrasound are good at predicting clinical numbers, but they often predict them for the wrong reasons. A model asked to estimate how strongly the heart pumps may instead learn from the text that the ultrasound machine burns into the image, or from the scanner's gain settings, because these happen to correlate with the label. The fix this note proposes is simple to state. We give a factorized video network two output heads. The first summarizes what the video looks like. The second summarizes how it moves, and is built so that it cannot see anything that stays still. We then train each head, without labels, so that its neighborhoods match a separate similarity graph: one graph built from appearance and one built from motion. The training loss is the one used by UMAP (Uniform Manifold Approximation and Projection, a method that lays out data so that points that are near neighbors in the original space stay near neighbors in the layout). Here that loss is used as a training objective inside the network rather than as a plot drawn afterwards.
The backbone is a factorized 3D convolution network, (2+1)D. Such a network splits each space-time filter into a 2D spatial filter followed by a 1D temporal filter. A Closer Look at Spatiotemporal Convolutions for Action Recognition showed this split trains better than full 3D filters. Yet every such network ends by averaging over space and time into one vector, which mixes appearance and motion back together. Our bet is that two changes give a pretrained representation in which static confounds cannot reach the motion head, and in which appearance-driven and motion-driven clinical variables can be read from separate coordinates. The first change is a motion head that is blind to static content by construction. The second is a pair of UMAP graph losses that tells each head what "similar" should mean.
Core Claim
A (2+1)D network has a useful property: it treats time the same way at every frame. If an image region does not change over time, such as a burned-in text overlay, its deep features are nearly constant over time, apart from edge effects of padding. Our motion head exploits this. It takes the final feature map before pooling and computes the magnitude of its Fourier transform along the time axis. It discards the zero-frequency term, the part of the signal that does not change at all. It then pools over space. The result does not depend on where the clip starts within the heart cycle, and it contains nothing from content that does not move.
The appearance head is ordinary average pooling over time. Each head $h \in \{s, t\}$, for spatial and temporal, maps a clip $x_i$ to an embedding $f_h(x_i)$. That embedding is trained against a fuzzy neighbor graph $P^h$ computed from cheap, label-free descriptors of the input. The objective is
$$L = \sum_{h \in \{s,t\}} \sum_{i \neq j} \Big[ -P^h_{ij}\log q^h_{ij} - (1-P^h_{ij}) \log(1-q^h_{ij}) \Big] + \lambda \sum_{(i,j) \in \mathcal{N}_s \setminus \mathcal{N}_t} -\log(1-q^t_{ij}).$$
Here $q^h_{ij} = (1+\|f_h(x_i)-f_h(x_j)\|^2)^{-1}$ is the Cauchy similarity between two embeddings. $\mathcal{N}_s$ and $\mathcal{N}_t$ are the edge sets of the appearance and motion graphs. $\lambda$ weights the last term. That term explicitly pushes apart, in the motion head, clips that look alike but do not move alike. The sum is optimized with UMAP's negative sampling.
The appearance graph $P^s$ comes from a frozen ImageNet 2D encoder applied to the temporal median frame of each clip. The motion graph $P^t$ comes from motion descriptors computed on a coarse 8×8 grid. For each cell we take the temporally low-passed frame-difference energy and compute its normalized power spectrum. Low-passing suppresses speckle decorrelation, the frame-to-frame flicker of ultrasound noise. Static overlays contribute exactly zero to these descriptors.
What would be new is the combination: input-level graphs that are factorized by construction, matched by heads that are factorized by construction, inside a factorized backbone. Prior work does any one of these only post hoc or not at all.
Relation to Prior Work
Three papers motivate the gap. CoReEcho trains a hybrid convolution-transformer encoder with a ranking-based contrastive loss so that videos with similar ejection fraction (EF, the fraction of blood the heart pumps out per beat) embed nearby. It documents that end-to-end video models latch onto overlays and acquisition clutter. However, it checks continuity only with a post-hoc t-SNE plot and keeps a single entangled embedding. We instead impose geometry during pretraining, without labels, and separate it into two factors. Deep Learning Prediction of Biomarkers from Echocardiogram Videos uses a (2+1)D-ResNet34 and shows, by hand-designed input blinding (single frames, contour masks), that anemia depends on texture while BNP, a blood marker of heart failure, depends on motion. We aim to recover that split from the model's own coordinates rather than from manual masking. The authors also pool into one vector, which is precisely what we replace.
Two papers supply the method. Contrastive learning unifies t-SNE and UMAP shows that UMAP's loss is negative sampling, an unnormalized relative of contrastive estimation with a fixed implicit normalization. It also shows that the Cauchy kernel removes instability at zero distance. We adopt the Cauchy kernel. We also treat the strength of repulsion as a dial to sweep, because the UMAP tutorial and survey warns that practical UMAP over-weights repulsion and tears continuous manifolds into clusters. That is a real threat when the target variable, EF, is continuous. The survey also describes Parametric UMAP, which trains a network on this loss for static vectors. We move that idea into a video backbone and give it two graphs.
Two papers frame the geometry and constrain our claims. Product Manifold Learning separates independent factors offline by spectral analysis of a product manifold. Learning disentangled representations via product manifold projection learns projections onto factor subspaces from weakly supervised pairs. Neither touches video or convolutional features. Both assume the factors are truly independent, which appearance and motion in echocardiography are not: a dilated heart also moves less. We therefore do not claim full independence. We claim only that static content is excluded from the motion head and that cross-leakage is measurable.
Research Program
The work has three parts.
The first part is building and validating the two graphs on unlabeled clips from EchoNet-Dynamic. We test the graphs before any network is trained. The question is whether motion-graph neighborhoods track EF better than appearance-graph neighborhoods do, and whether appearance-graph neighborhoods track overlay and vendor identity better than motion-graph neighborhoods do. If the motion descriptors do not carry EF information at the graph level, the program stops here.
The second part is pretraining an R(2+1)D backbone with the two heads and the loss above, sweeping $\lambda$ and the negative-sampling rate.
The third part is evaluation. We use linear probes and light fine-tuning on EF, transfer to myocardial infarction detection on HMC-QU, and controlled confound tests. If the Stanford/Cedars-Sinai EchoNet-Labs cohort is accessible, we also probe biomarkers per head. Since that cohort may not be public, it is treated as an optional extension.
Baselines
The naive reference point is an R(2+1)D network trained end to end to regress EF with plain average pooling, no pretraining loss, and the same augmentation. This is the setting that CoReEcho reports to be vulnerable to spurious features. For external calibration, Video-based AI for beat-to-beat assessment of cardiac function reports an EF mean absolute error (MAE) of 4.1% internally and 6.0% on an external set, and Ultrasound Video Transformers reports an MAE of 5.95.
The strongest existing method is CoReEcho, which reports an EF MAE of 3.90 and an R² of 82.44 on EchoNet-Dynamic, and 90.64% accuracy for infarction detection on HMC-QU. Because CoReEcho uses a different backbone, we run two versions. In the first, we reproduce its Rank-n-Contrast loss on our exact R(2+1)D backbone, with the same data split, epochs, and hyperparameter search budget as our method. This isolates the loss from the architecture. In the second, we quote the published figures for the original model. Beating CoReEcho means one of two things. Either our model reaches lower MAE than the matched Rank-n-Contrast model at full labels, or it matches it at full labels while degrading less under the confound tests and at 1% and 10% of labels.
A label-free competitor is masked-autoencoder pretraining using the public E-ViM3 codebase, from the Mamba-3D masked autoencoder paper ( E-ViM 3 : Mamba-3D as Masked Autoencoders for Accurate and Data-Efficient Analysis of Medical Ultrasound Videos). No reported EF result for it was found on Lacuna, so we run it ourselves under the same probe protocol.
Two ablations isolate the key ingredient. The first keeps the spectral motion head but trains it with a standard single-graph SimCLR-style loss instead of the motion graph. This asks whether the head architecture alone does the work. The second keeps both graphs but uses ordinary average pooling for both heads. This asks whether the loss alone does the work.
Why not just…?
Why not just mask or crop the overlay text before training? Masking removes one known confound, but CoReEcho's description of the failure covers acquisition clutter more broadly, and gain settings or patient-dependent brightness cannot be cropped out. Our confound tests therefore include static perturbations that no mask could anticipate. These are synthetic overlays placed at random positions, global gain shifts, and static vignettes, each correlated with EF in the training set and decorrelated at test time. Cropping is included as a preprocessing baseline under these same tests.
Why not just apply Rank-n-Contrast to an R(2+1)D backbone? This is the most serious threat, and the matched-backbone baseline exists to settle it. If label-supervised ranking matches us on accuracy and on robustness to confounds at every label budget, then the dual-graph pretraining earns nothing for EF. What would remain is the per-head attribution analysis, and we would report that as the only remaining contribution.
Why not just feed frame differences to the network as a motion-only input? This could plausibly match the motion head on EF, so we test it directly with an R(2+1)D trained on low-passed frame differences. Our expectation is that it loses accuracy because ultrasound frame differences are dominated by speckle. Even if it ties on EF, it yields no appearance coordinates. It therefore cannot reproduce the texture-versus-motion biomarker split that Deep Learning Prediction of Biomarkers from Echocardiogram Videos found. Whether two heads are needed is answered by the HMC-QU and biomarker probes, not by EF alone.
Why not just run UMAP on the pooled features after training? That is the current practice. It cannot change what the network learned, and a single pooled vector has already mixed the factors. Our "average pooling, both graphs" ablation and the end-to-end baseline followed by post-hoc UMAP show how much is lost.
Proposed Experiments
On EchoNet-Dynamic we report EF MAE and R² for linear probes at 1%, 10%, and 100% of labels, and for full fine-tuning. The EF regressor reads from the motion head, the appearance head, or both.
The decisive checks are the following. First, under the static-confound tests, the motion head's EF predictions should shift by much less than those of the end-to-end baseline and the matched Rank-n-Contrast model. If they shift comparably, the central claim is false.
Second, under frame shuffling, the appearance head's embedding should barely move while the motion head's EF accuracy collapses. Under reversing time, the motion head should be largely unchanged, since magnitude spectra ignore phase.
Third, we measure cross-leakage by how well vendor or overlay identity can be decoded from the motion head. It should be near chance.
Fourth, on HMC-QU, transfer accuracy is compared against CoReEcho's reported 90.64% and against our matched baseline.
Fifth, we sweep the repulsion strength. If EF continuity in the motion head breaks into clusters at UMAP-like settings and recovers only near t-SNE-like settings, we report that as a finding about which neighbor-embedding regime suits continuous clinical variables. We judge continuity by the Spearman correlation between EF and position along the head's principal curve.
Finally, if EchoNet-Labs is available, the per-head probes should reproduce the split found by input blinding: hemoglobin decoded better from the appearance head, BNP better from the motion head.
References
- A Closer Look at Spatiotemporal Convolutions for Action Recognition
- CoReEcho: Continuous Representation Learning for 2D+Time Echocardiography Analysis
- Deep Learning Prediction of Biomarkers from Echocardiogram Videos
- Contrastive learning unifies t-SNE and UMAP
- Uniform Manifold Approximation and Projection (UMAP) and its Variants: Tutorial and Survey
- Product Manifold Learning
- Learning disentangled representations via product manifold projection
- Video-based AI for beat-to-beat assessment of cardiac function
- Ultrasound Video Transformers for Cardiac Ejection Fraction Estimation
- E-ViM 3 : Mamba-3D as Masked Autoencoders for Accurate and Data-Efficient Analysis of Medical Ultrasound Videos
- E-ViM 3 : Mamba-3D as Masked Autoencoders for Accurate and Data-Efficient Analysis of Medical Ultrasound Videos