Diagnosis Labels as Weak Supervision for Cardiac Motion

Cardiologists tell heart diseases apart mostly by how the heart moves. A ventricle that is enlarged and moves weakly everywhere suggests one disease; one that is enlarged but still moves in some regions suggests a scarred infarct. Current automatic systems recover this motion poorly. They segment each frame of a cardiac MRI video independently, compute chamber volumes over time, and feed those numbers to a separate classifier. No pixel is tracked from frame to frame, and nothing the classifier learns ever flows back to improve the segmentation. Deformable registration can track tissue. It estimates a dense displacement field mapping each point in one frame to its position in another. The best cardiac registration models, however, stop at producing that field and never diagnose anything.

We propose to train registration, segmentation and diagnosis together. The classifier is forced to see the heart only through a small set of physically meaningful motion quantities computed from the registration field: regional volume curves, regional strain (how much tissue stretches or shortens), and local volume change. Because this "motion bottleneck" is differentiable, a patient-level diagnosis label becomes a training signal for the motion and segmentation models. Diagnosis labels are cheap and plentiful, while dense manual contours are expensive and scarce. The claim is therefore specific: when only a handful of patients have hand-drawn contours but all have a diagnosis, routing diagnosis gradients through a physical motion bottleneck should improve segmentation and diagnosis. It should do so without corrupting the motion estimates, and that last condition is something we will measure rather than assume.

Core Claim

Joint registration and segmentation already exists. DeepAtlas trains a registration network and a segmentation network that supervise each other in alternating steps on knee and brain MRI. SDDIR uses predicted cardiac masks to compose separate deformation fields per chamber. This preserves sliding motion at boundaries and reaches 84.46% average Dice on UK Biobank. Neither predicts disease.

On the diagnosis side, the ACDC two-stage pipeline reaches 92% five-class test accuracy from handcrafted volume features. It uses no tracking and has no backward path from classifier to segmenter, and the file notes that it has trouble separating dilated cardiomyopathy from myocardial infarction. ViACT classifies cardiac amyloidosis from tracked myocardial points, but the tracking comes offline from external software.

Our proposal differs from all four in one specific way. The diagnosis loss reaches the registration and segmentation weights, and it reaches them only through an interpretable, low-dimensional physical interface. That interface is both the interpretability mechanism and the guard against the classifier exploiting arbitrary image features.

Research Program

The model is a single cine-MRI encoder with three heads. The segmentation head predicts left-ventricle cavity, myocardium and right-ventricle masks for every frame. The registration head predicts frame-to-frame displacement fields. Following SDDIR, these fields are composed from per-region sub-fields using the predicted masks, so motion can be discontinuous at chamber boundaries.

We close SDDIR's open loop with a consistency term. The end-diastole mask, warped by the predicted field, must match the predicted end-systole mask. This means registration errors send gradients into the segmenter and segmentation errors send gradients into the registration, rather than the one-way dependency SDDIR reports, where registration collapses once segmentation Dice falls below roughly 70–76%.

From the warped masks and the field we compute the bottleneck vector $m$ with fixed, differentiable formulas. It contains per-region volume curves over the cycle, mean radial and circumferential strain in a fixed number of myocardial sectors, and the distribution of the Jacobian determinant (the local volume change of the field) within the myocardium. A small classifier maps $m$ to one of the five ACDC classes. The training objective is

$$ \mathcal{L} = \mathcal{L}_{seg}\,\mathbb{1}[\text{contours available}] + \mathcal{L}_{sim} + \alpha\,\mathcal{L}_{smooth} + \beta\,\mathcal{L}_{cons} + \gamma\,\mathcal{L}_{diag}(f(m)), $$

The terms are defined as follows:

  • $\mathcal{L}_{seg}$ is the supervised segmentation loss, applied only to patients with contours.
  • $\mathcal{L}_{sim}$ is image similarity between the warped and target frames.
  • $\mathcal{L}_{smooth}$ is the within-region smoothness penalty.
  • $\mathcal{L}_{cons}$ is the warped-mask consistency term.
  • $\mathcal{L}_{diag}$ is cross-entropy of the classifier $f$ applied to the bottleneck $m$.
  • $\alpha, \beta, \gamma$ are weights.

The pixel-dense losses and the single categorical loss produce gradients of very different size and direction. We therefore compare static weights against RotoGrad. RotoGrad rescales each task's gradient according to how fast that task is converging and rotates the shared features per task to reduce directional conflict. It reports gains over plain summed losses on NYUv2 and CelebA. Because full rotations are costly in high dimensions, we apply them to the pooled encoder bottleneck only.

The main risk is that diagnosis gradients bend the field so that strain encodes the class label rather than true motion. This would make the motion estimates more useful to the classifier and less correct as motion. The program treats this as a measured outcome.

Baselines

The naive reference is the ACDC pipeline itself: per-frame segmentation, handcrafted volume features, and a tree or MLP ensemble. The ACDC pipeline paper reports 92% test accuracy (94% in cross-validation) and test Dice of 0.950 left-ventricle cavity, 0.923 right ventricle and 0.911 myocardium with all 100 training patients contoured. We re-run it at each annotation budget with the same segmentation backbone, so it is not credited with a stronger network than ours.

The strongest competitor on the actual question is a two-stage system. First, a DeepAtlas-style joint registration and segmentation model, with SDDIR's region-composed fields and our consistency term, is trained on the available contours. Its weights are then frozen, the identical bottleneck $m$ is computed, and a classifier is trained on it with all diagnosis labels. This baseline has every ingredient except diagnosis gradients reaching the motion model, so it isolates the key ingredient.

No reported result on ACDC diagnosis was found on Lacuna for DeepAtlas or SDDIR. For registration quality, SDDIR's reported 84.46% UK Biobank Dice against 74.18% for VoxelMorph is the reference. A third baseline is a direct spatio-temporal video classifier on raw cine frames, given the same tuning budget, to measure what the bottleneck costs or gains.

All systems share the same data splits, encoder size, augmentation and hyperparameter search budget (the same number of trials per method), with five seeds each. Beating the frozen two-stage system means a statistically supported gain in segmentation Dice or diagnostic accuracy at low annotation budgets, with registration plausibility no worse.

Why not just…?

Why not just freeze a good registration model and add strain features to the existing classifier? This is the most serious threat, and we expect it to match our model when all 100 patients are contoured. Its upstream models then receive no benefit from diagnosis labels. The frozen two-stage baseline is exactly this system. If it matches the joint model at 5 and 10 contoured patients too, the claim is false and we will report that diagnosis supervision adds nothing beyond better features.

Why not just sum the losses with hand-tuned weights? It may suffice. Blasto-Net (paper) combines segmentation and clinical prediction heads with static weights and still beats single-task models on blastocyst images. Against that, DeepAtlas needed a 1:20 alternating schedule and pre-training to keep joint training stable. We run static weights, the alternating schedule, magnitude-only balancing and full RotoGrad side by side. If static weights tie, we drop RotoGrad.

Why not just classify directly from the video? A video classifier can use any pixel, so it may score well on ACDC by exploiting appearance. ViACT reports that restricting attention to anatomy halved compute versus an unconstrained video transformer while classifying well, but it did not test ACDC. Our comparison shows whether the bottleneck costs accuracy. The out-of-distribution test below shows whether the bottleneck buys robustness.

Proposed Experiments

Annotation sweep. We train on ACDC with contours for 5, 10, 25 and 100 training patients and diagnosis labels for all 100. We report Dice and surface distance at end-diastole and end-systole, and five-class accuracy with per-class confusion, focusing on the infarction-versus-dilated pair.

Statistical treatment. The 50-patient test set means one patient equals two accuracy points. We therefore use repeated cross-validation with bootstrap intervals and treat differences under three patients as ties.

Motion fidelity. This is the decisive check against shortcut learning. We compare our model with the frozen baseline on:

  • warped-mask Dice across the cycle;
  • the fraction of voxels with non-positive Jacobian, which indicates folding;
  • agreement of predicted ejection fraction and myocardial mass with values from the manual contours.

If diagnosis training improves accuracy but worsens these measures, the field is encoding labels rather than motion.

Transfer. We apply segmentation and registration trained on ACDC to M&M, a multi-vendor cardiac MRI set used here for domain shift. If access permits, we also evaluate on UK Biobank registration against SDDIR's 84.46%.

Gradient ablation. Finally, we vary $\gamma$ from zero upward to trace the trade-off between diagnostic gain and motion fidelity. The proposal succeeds only if some setting improves low-label segmentation or diagnosis while holding motion fidelity at the frozen baseline's level.