Learning a Generative Motion Model from Image Sequences based on a Latent Motion Matrix

Julian Krebs, Hervé Delingette, Nicholas Ayache, Tommaso Mansi

Abstract

We propose to learn a probabilistic motion model from a sequence of images for spatio-temporal registration. Our model encodes motion in a low-dimensional probabilistic space—the motion matrix—which enables various motion analysis tasks such as simulation and interpolation of realistic motion patterns, allowing for faster data acquisition and data augmentation. More precisely, the motion matrix allows for the transport of recovered motion from one subject to another, simulating, for example, a pathological motion in a healthy subject without the need for inter-subject registration. The method is based on a conditional latent variable model that is trained using amortized variational inference. This unsupervised generative model follows a novel multivariate Gaussian process prior and is applied within a temporal convolutional network, leading to a diffeomorphic motion model. Temporal consistency and generalizability are further improved by applying a temporal dropout training scheme. Applied to cardiac cine-MRI sequences, we show improved registration accuracy and spatio-temporally smoother deformations compared to three state-of-the-art registration algorithms. Besides, we demonstrate the model's applicability for motion analysis, simulation, and super-resolution by an improved motion reconstruction from sequences with missing frames compared to linear and cubic interpolation.

Introduction

In clinical cardiology, tracking the motion of the heart is essential for assessing cardiac function and diagnosing pathologies. Traditional image registration methods often treat sequences as a series of independent pairs, which can result in temporally inconsistent or "jittery" motion estimates. While deep learning has accelerated registration, many existing models lack a structured generative space that allows for the analysis and simulation of motion independent of a patient's specific anatomy. This paper introduces a framework that not only tracks motion with high accuracy but also learns a generative "latent motion matrix" capable of representing complex temporal dynamics.

Methodology

The core of the proposed method is an unsupervised Conditional Latent Variable Model designed to decouple anatomy from motion.

The proposed architecture featuring an encoder-decoder structure with a Temporal Convolutional Network (TCN) and a latent motion matrix.

The Latent Motion Matrix and GP Prior

Rather than encoding a sequence into a single vector, the model utilizes a latent motion matrix ($z$), where rows represent latent dimensions and columns represent time steps. To ensure temporal smoothness, the authors employ a Gaussian Process (GP) prior across the temporal dimension. Unlike standard VAEs that assume independence between latent variables, the GP prior—specifically using a Cauchy kernel—enforces continuity and allows for long-range temporal influence, resulting in physically realistic motion trajectories.

Temporal Convolutional Networks (TCN) and Dropout

To map image sequences to this latent space, the framework uses a TCN. TCNs offer a flexible receptive field and superior stability for the relatively short sequences typical of cardiac MRI compared to Recurrent Neural Networks (RNNs). A key innovation is Temporal Dropout (TD): during training, the model randomly omits frames, forcing the network to learn how to interpolate missing motion. This self-supervised mechanism enables the model to perform temporal super-resolution and handle missing data effectively.

Diffeomorphic Mapping

To maintain physical plausibility (preventing tissue "folding"), the decoder uses a stationary velocity field (SVF) parameterization with a scaling-and-squaring layer. This ensures that the resulting deformations are diffeomorphic and invertible.

Experiments and Results

The model was validated on 334 cardiac cine-MRI sequences from the ACDC 2017 Challenge and the MD-Paedigree project. The dataset included a variety of cardiac pathologies as well as healthy subjects.

Quantitative evaluation comparing the proposed method against state-of-the-art registration algorithms across multiple metrics including DICE, Hausdorff Distance, and temporal gradients.

Registration Performance

The proposed model achieved a state-of-the-art DICE score of 85.2%, significantly outperforming traditional methods like SyN (82.7%) and 4D-Elastix (83.7%). Beyond accuracy, the model produced much smoother deformations; temporal gradients were reduced by more than 60% compared to 4D-Elastix, indicating a more natural representation of cardiac contraction and relaxation.

Motion Transport and Simulation

By conditioning the decoder on the initial frame ($I_0$), the latent matrix $z$ captures motion characteristics independently of anatomy. This enables Motion Transport: a pathological motion signature (e.g., from a patient with Dilated Cardiomyopathy) can be applied to a healthy subject's heart to simulate how that specific anatomy would behave under disease. Furthermore, the model demonstrated superior performance in reconstructing full cardiac cycles from sparse data, outperforming standard linear and cubic interpolation methods.

Conclusion

This work presents a robust generative framework for cardiac motion modeling. By integrating a GP prior with a TCN-based CVAE, the model achieves superior registration accuracy and temporal consistency. The ability to decouple motion from anatomy opens new avenues for data augmentation and clinical simulation. Future research will focus on extending this model to respiratory motion in the lungs and exploring its utility in automated disease classification.

Summary updated

Abstract

We propose to learn a probabilistic motion model from a sequence of images for spatio-temporal registration. Our model encodes motion in a low-dimensional probabilistic space - the motion matrix - which enables various motion analysis tasks such as simulation and interpolation of realistic motion patterns allowing for faster data acquisition and data augmentation. More precisely, the motion matrix allows to transport the recovered motion from one subject to another simulating for example a pathological motion in a healthy subject without the need for inter-subject registration. The method is based on a conditional latent variable model that is trained using amortized variational inference. This unsupervised generative model follows a novel multivariate Gaussian process prior and is applied within a temporal convolutional network which leads to a diffeomorphic motion model. Temporal consistency and generalizability is further improved by applying a temporal dropout training scheme. Applied to cardiac cine-MRI sequences, we show improved registration accuracy and spatio-temporally smoother deformations compared to three state-of-the-art registration algorithms. Besides, we demonstrate the model's applicability for motion analysis, simulation and super-resolution by an improved motion reconstruction from sequences with missing frames compared to linear and cubic interpolation.
BibTeX
@article{krebs2021learning,
  author = {Julian Krebs and Hervé Delingette and Nicholas Ayache and Tommaso Mansi},
  title = {{Learning a Generative Motion Model from Image Sequences based on a Latent Motion Matrix}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2021},
  url = {https://openreview.net/forum?id=G_SNyq-24C},
}
Figure