Latent linear dynamics in spatiotemporal medical data

Niklas Gunnarsson, Jens Sjölund, Thomas B. Schön

Abstract

Spatiotemporal imaging has applications in e.g. cardiac diagnostics, surgical guidance, and radiotherapy monitoring. In this paper, we explain the temporal motion by identifying the underlying dynamics, only based on the sequential images. Our dynamical model maps the inputs of observed high-dimensional sequential images to a low-dimensional latent space wherein a linear relationship between a hidden state process and the lower-dimensional representation of the inputs holds. For this, we use a conditional variational auto-encoder (CVAE) to nonlinearly map the higher-dimensional image to a lower-dimensional space, wherein we model the dynamics with a linear Gaussian state-space model (LG-SSM). The model, a modified version of the Kalman variational auto-encoder, is end-to-end trainable, and the weights, both in the CVAE and LG-SSM, are simultaneously updated by maximizing the evidence lower bound of the marginal likelihood. In contrast to the original model, we explain the motion with a spatial transformation from one image to another. This results in sharper reconstructions and the possibility of transferring auxiliary information, such as segmentation, through the image sequence. Our experiments, on cardiac ultrasound time series, show that the dynamic model outperforms traditional image registration in execution time, to a similar performance. Further, our model offers the possibility to impute and extrapolate for missing samples.

Introduction

Spatiotemporal imaging—like ultrasound or 4D-CT—is vital for cardiac diagnostics and guided surgery. However, analyzing these sequences is difficult because the motion is both high-dimensional and nonlinear. Traditional "image registration" methods (aligning one image to another) are often too slow for real-time use. Meanwhile, standard Deep Learning approaches like Variational Auto-encoders (VAEs) often produce "blurry" results because they attempt to predict every pixel's intensity directly. Furthermore, many existing models struggle to handle missing data (imputation) or predict future motion (extrapolation) effectively.

Methods

The authors propose a modified Kalman Variational Auto-encoder (KVAE). Their approach relies on two key components:

  1. Dimensionality Reduction (CVAE): A Conditional Variational Auto-encoder nonlinearly maps high-dimensional images into a small, manageable "latent space."
  2. Latent Dynamics (LG-SSM): Inside this latent space, the motion follows a Linear Gaussian State-Space Model. This allows for the use of classical tools like Kalman filters and Rauch-Tung-Striebel (RTS) smoothers to track the state of the system.
Model architecture and graphical representation.

Unlike a standard KVAE that reconstructs the image $y_t$ directly, this model predicts the spatial transformation ($\phi_t$) required to warp a reference image into the current frame. Because the model predicts how to move pixels rather than what color the pixels are, the resulting images remain sharp, and the model can automatically propagate a manual segmentation from the first frame through the entire sequence.

Experimental Results

The model was trained and evaluated on the EchoNet-Dynamic dataset, which contains 10,030 2D ultrasound echocardiogram time series. Each sequence was fixed to 50 time steps (roughly one cardiac cycle) with images resized to $112 \times 112$ pixels. The dataset includes expert human segmentations of the left ventricle, used to validate the accuracy of the learned transformations.

Key findings include:

  • Accuracy: The model achieved an average Dice score of 0.81, matching the performance of established clinical baselines like the iterative Demons algorithm.
  • Speed: The proposed model is significantly faster, processing an entire sequence in approximately 0.09 seconds on a GPU, compared to 1.64 seconds for the Demons algorithm on a CPU.
  • Functionality: The state-space framework allows the model to naturally handle missing data (imputation) and predict future heart positions (extrapolation), which is critical for compensating for latency in robotic surgery or radiotherapy.

Conclusion and Future Work

Modeling the displacement field rather than pixel intensity avoids the "averaging" effect of generative models and ensures anatomical preservation. While heart motion is complex in pixel space, it can be effectively modeled as a linear process in a properly learned latent space.

Future work will address current limitations, such as occasional physically implausible "warping" (non-positive Jacobians) in low-texture regions. The authors also aim to extend this framework from 2D ultrasound to 3D motion and explore more complex likelihood distributions to better capture spatial dependencies.

Summary updated

Abstract

Spatiotemporal imaging has applications in e.g. cardiac diagnostics, surgical guidance, and radiotherapy monitoring, In this paper, we explain the temporal motion by identifying the underlying dynamics, only based on the sequential images. Our dynamical model maps the inputs of observed high-dimensional sequential images to a low-dimensional latent space wherein a linear relationship between a hidden state process and the lower-dimensional representation of the inputs holds. For this, we use a conditional variational auto-encoder (CVAE) to nonlinearly map the higher-dimensional image to a lower-dimensional space, wherein we model the dynamics with a linear Gaussian state-space model (LG-SSM). The model, a modified version of the Kalman variational auto-encoder, is end-to-end trainable, and the weights, both in the CVAE and LG-SSM, are simultaneously updated by maximizing the evidence lower bound of the marginal likelihood. In contrast to the original model, we explain the motion with a spatial transformation from one image to another. This results in sharper reconstructions and the possibility of transferring auxiliary information, such as segmentation, through the image sequence. Our experiments, on cardiac ultrasound time series, show that the dynamic model outperforms traditional image registration in execution time, to a similar performance. Further, our model offers the possibility to impute and extrapolate for missing samples.
BibTeX
@inproceedings{gunnarsson2022latent,
  author = {Niklas Gunnarsson and Jens Sjölund and Thomas B. Schön},
  title = {{Latent linear dynamics in spatiotemporal medical data}},
  booktitle = {FUSION},
  year = {2022},
}
Figure