Unsupervised dynamic modeling of medical image transformation

Niklas Gunnarsson, Peter Kimstrand, Jens Sjölund, Thomas B. Schön

Abstract

Spatiotemporal imaging is essential for applications such as cardiac diagnostics, surgical guidance, and radiotherapy monitoring. This paper presents a method for explaining temporal motion by identifying underlying dynamics directly from sequential images. The proposed dynamical model maps high-dimensional observed images to a low-dimensional latent space where a linear relationship exists between a hidden state process and the lower-dimensional representation. This is achieved using a conditional variational auto-encoder (CVAE) for nonlinear mapping and a linear Gaussian state-space model (LG-SSM) for modeling dynamics. Unlike traditional models, this approach explains motion through spatial transformations, resulting in sharper reconstructions and enabling the transfer of auxiliary information like segmentations. Experiments on cardiac ultrasound time series demonstrate that this model matches the performance of traditional image registration while offering significant speed improvements and the ability to impute or extrapolate missing samples.

Introduction

Capturing how anatomy moves over time is critical for medical diagnostics and intervention. However, analyzing this motion is computationally challenging. Traditional "image registration" methods—which align one image to another—are often slow and iterative, making them poorly suited for real-time clinical applications.

While deep learning models like Variational Auto-Encoders (VAEs) can compress images into simpler latent representations, they often produce blurry outputs. Furthermore, modeling the complex, deformable nature of human anatomy without relying on external markers is difficult. This research introduces an unsupervised framework that learns these dynamics by modeling spatial transformations in a low-dimensional latent space, ensuring fast, sharp, and physically plausible motion estimation.

Methodology

The researchers propose a Deep State-Space Model (DSSM) that operates on image transformations rather than raw pixels. The architecture consists of two main components trained end-to-end:

  1. Conditional Variational Auto-Encoder (CVAE): This component nonlinearly maps high-dimensional images ($y_t$) into a low-dimensional latent space ($x_t$). Crucially, the decoder estimates a spatial transformation ($\phi_t$) which is then applied to a reference "moving" image ($y_M$) to produce the final output. This avoids the common "blurry VAE" problem because the final image is a warped version of a high-quality original.
  2. Linear Gaussian State-Space Model (LG-SSM): Inside the latent space, the model assumes motion follows linear dynamics governed by a hidden state ($z_t$). This allows the use of classical Kalman filtering and smoothing to handle noise and temporal uncertainty.
The architecture (a) and graphical model (b) of the Deep State-Space Model, illustrating the relationship between the CVAE and the LG-SSM.

By maximizing the Evidence Lower Bound (ELBO), the model simultaneously learns how to compress the images and how to describe the underlying "physics" of the motion between them.

Experimental Results

The model was evaluated using the EchoNet-Dynamic dataset, comprising 10,030 2D ultrasound echocardiogram time series.

Key Performance Metrics

  • Accuracy: The model achieved an average Dice score of 0.81, matching the performance of the traditional iterative Demons algorithm in tracking the left ventricle.
  • Speed: The proposed model is significantly faster. On a GPU, it processed a sequence in 0.09 seconds, compared to 1.64 seconds for the Demons algorithm—a nearly 18x speedup that enables near real-time use.
  • Topology Preservation: By applying a Gaussian kernel to regularize the displacement field, the researchers reduced mathematically impossible tissue "folds" (non-positive Jacobians) from 2.7% to just 0.5%.

Practical Applications

Because the model learns a transformation, it can automatically propagate auxiliary data. For instance, if a clinician segments the heart in the first frame, the model can propagate that segmentation through the entire sequence automatically. Additionally, the state-space structure allows the model to handle missing data (imputation) or predict future frames (extrapolation), which is vital for compensating for system latency in robotic surgery.

Discussion and Future Work

While the model shows great promise, it is currently limited to 2D ultrasound. Extending this framework to 3D motion is a primary goal for future research. Additionally, the model occasionally produces implausible transformation vectors in "zero-intensity" regions (black space) where there is no signal to guide registration. The authors suggest that using more complex likelihood structures, such as a Laplacian precision matrix, could further improve uncertainty estimation and spatial correlation handling.

Conclusion

This work demonstrates that complex anatomical motion can be effectively captured by combining deep generative models with classical state-space dynamics. By focusing on transformations in a latent space, the DSSM provides a robust, unsupervised solution for real-time spatiotemporal medical imaging.

Summary updated

Abstract

Spatiotemporal imaging has applications in e.g. cardiac diagnostics, surgical guidance, and radiotherapy monitoring, In this paper, we explain the temporal motion by identifying the underlying dynamics, only based on the sequential images. Our dynamical model maps the inputs of observed high-dimensional sequential images to a low-dimensional latent space wherein a linear relationship between a hidden state process and the lower-dimensional representation of the inputs holds. For this, we use a conditional variational auto-encoder (CVAE) to nonlinearly map the higher-dimensional image to a lower-dimensional space, wherein we model the dynamics with a linear Gaussian state-space model (LG-SSM). The model, a modified version of the Kalman variational auto-encoder, is end-to-end trainable, and the weights, both in the CVAE and LG-SSM, are simultaneously updated by maximizing the evidence lower bound of the marginal likelihood. In contrast to the original model, we explain the motion with a spatial transformation from one image to another. This results in sharper reconstructions and the possibility of transferring auxiliary information, such as segmentation, through the image sequence. Our experiments, on cardiac ultrasound time series, show that the dynamic model outperforms traditional image registration in execution time, to a similar performance. Further, our model offers the possibility to impute and extrapolate for missing samples.
BibTeX
@inproceedings{gunnarsson2022unsupervised,
  author = {Niklas Gunnarsson and Peter Kimstrand and Jens Sjölund and Thomas B. Schön},
  title = {{Unsupervised dynamic modeling of medical image transformation}},
  booktitle = {FUSION},
  year = {2022},
}
Figure