Latent-space disentanglement with untrained generator networks for the isolation of different motion types in video data

Abdullah Abdullah, Martin Höller, Karl Kunisch, Malena Sabaté Landman

TL;DR

This paper introduces a novel method for isolating different types of motion within a single video, such as cardiac and respiratory movements, by leveraging untrained generator networks and latent space disentanglement. This approach allows for the creation of videos showing only a specific motion type without requiring pre-training on large datasets, offering a more general and efficient solution for motion analysis in various applications.

What Problem Does This Paper Address?

Analyzing dynamic processes in video data is crucial across many fields, from medical imaging to autonomous driving. A significant challenge arises when multiple types of motion are present simultaneously, complicating the analysis of a specific motion of interest. For instance, in medical imaging like MRI, a video might capture both the beating heart (cardiac motion) and breathing (respiratory motion). Isolating one of these motions, say cardiac motion, is essential for accurate analysis, but existing methods often struggle with this task. While techniques like image registration can align frames, they don't inherently decompose complex motions into their constituent types. Existing approaches for motion isolation often rely on specific assumptions about the motion's structure or require extensive prior knowledge and training data, limiting their applicability. This paper aims to address this gap by proposing a method that can efficiently disentangle and isolate different motion types from a single video, even with complex, non-linear movements, without needing pre-training.

Methods

The core of the proposed method lies in using untrained generator networks combined with a specific latent space disentanglement technique.

  1. Untrained Generator Networks (Deep Image Prior): The paper adapts the "deep image prior" concept, where a generator network is trained to reconstruct a given image or sequence of images. Crucially, this generator is untrained in the traditional sense; its parameters are learned directly from the data itself, without relying on pre-existing datasets. The generator network maps a latent space vector ($z$) to an image ($x = G(z)$).
  2. Latent Space Disentanglement: The key innovation is how the latent space ($z$) is structured and optimized. The latent space is split into:
  3. A static component ($z^0$) representing the underlying structure of the image.
  4. Multiple dynamic components ($z^1, z^2, \dots, z^m$), each intended to capture a different type of motion.

    The disentanglement is achieved by providing one-dimensional "motion triggers" (external information) for all but one of the motion types. For example, if isolating cardiac motion from respiratory motion, the respiratory motion trigger (e.g., from chest displacement) is provided, while the cardiac motion trigger is treated as an unknown optimization variable. This forces the network to learn to represent the provided motion trigger in one latent variable and the unknown motion in another, effectively disentangling them.

  5. Joint Optimization: The entire system (generator network parameters and the unknown latent variables) is optimized jointly to reconstruct the observed video data. This is formulated as a minimization problem where the difference between the observed data and the generator's output is minimized.

  6. Motion Isolation: Once the optimization is complete, the disentangled latent variables can be used to isolate motion. By fixing the latent variables corresponding to all but one motion type to a reference frame (e.g., a static image or a frame where the other motion is frozen), and allowing only the latent variable for the desired motion to vary, the generator can produce a video showing only that specific motion.

Datasets

The paper evaluates its method on:

  • Synthetic Data: A synthetic example with two nested disks exhibiting both respiratory and cardiac motion was generated.
  • Real Cardiac MR Images: Four-chamber and short-axis view cine MR images were used. These real images were augmented with synthetic respiratory motion to create ground truth data for evaluation. The authors note that the real MR data comes from the ISMRM reconstruction challenge 2014.

No new benchmarks were introduced; the focus was on applying the method to existing data types.

Key Insights and Ideas

  • Untrained Generators for Motion Disentanglement: The paper demonstrates that untrained generator networks, typically used for image reconstruction, can be effectively repurposed for disentangling and isolating motion components in dynamic data. This bypasses the need for large, pre-labeled datasets.
  • Minimal External Information for Powerful Disentanglement: The ability to disentangle complex, non-linear motions by providing only one-dimensional "triggers" for all but one motion type is a significant insight. This suggests that even limited external information can guide the network to learn meaningful representations of underlying dynamics.
  • Generalizability Beyond Medical Imaging: While demonstrated on cardiac and respiratory motion in MR images, the method's reliance on latent space disentanglement and generator networks suggests broad applicability to other domains involving dynamic video data (e.g., robotics, biomechanics).
  • Implicit Motion Modeling: The method does not require explicit mathematical models of the motions being isolated. The generator network learns these motion dynamics implicitly from the data and the disentanglement constraints.

Key Results

  • Successful Motion Isolation: The proposed method successfully isolates cardiac and respiratory motion in both synthetic and real MR image data.
  • Sharper Reconstructions: Compared to a baseline ICA (Independent Component Analysis) method, the proposed approach yields sharper reconstructions of the isolated motion types, as evidenced by visual inspection and error metrics.
  • Quantitative Performance: Table 1 shows quantitative error measures (median and MAD) for various experiments. For instance, in the phantom example, the error for isolated cardiac motion is around 1.06e-02 (median). While specific comparisons to state-of-the-art are not always direct due to the novelty of the task, the results are presented as competitive and superior to the ICA baseline.
  • Efficiency: The method is computationally efficient, with experiments on synthetic data taking around 19 seconds and on real MR data taking up to 3.8 minutes on a high-end workstation.

Limitations and Future Work

  • Non-convexity and Initialization Dependence: The optimization problem is non-convex, meaning the quality of the solution can depend on the initial random seeds used for training. The authors mitigate this by selecting the best result from multiple runs and suggest restarting strategies.
  • Artifacts in Breathing Motion Isolation: Some artifacts were observed in the isolation of breathing motion, which the authors attribute to suboptimal solutions of the minimization problem. They suggest that improved convergence through favorable initializations can reduce these artifacts.
  • No Explicit Motion Information: The method does not provide explicit motion fields or parameters, which might be a limitation for applications requiring such detailed information. However, this also allows for greater generality.
  • Future Work: The authors suggest exploring more advanced disentanglement schemes, potentially by modifying the loss function or adding constraints on the latent space variables. They also believe the method is well-suited for applications requiring dynamic inverse problems and the separation of independent motions.

Summary updated

Abstract

Isolating different types of motion in video data is a highly relevant problem in video analysis. Applications can be found, for example, in dynamic medical or biological imaging, where the analysis and further processing of the dynamics of interest is often complicated by additional, unwanted dynamics, such as motion of the measurement subject. In this work, it is empirically shown that a representation of video data via untrained generator networks, together with a specific technique for latent space disentanglement that uses minimal, one-dimensional information on some of the underlying dynamics, allows to efficiently isolate different, highly non-linear motion types. In particular, such a representation allows to freeze any selection of motion types, and to obtain accurate independent representations of other dynamics of interest. Obtaining such a representation does not require any pre-training on a training data set, i.e., all parameters of the generator network are learned directly from a single video.
BibTeX
@inproceedings{abdullah2023latent,
  author = {Abdullah Abdullah and Martin Höller and Karl Kunisch and Malena Sabaté Landman},
  title = {{Latent-space disentanglement with untrained generator networks for the isolation of different motion types in video data}},
  booktitle = {SSVM},
  year = {2023},
  url = {https://openreview.net/forum?id=sLyL6zpB9a},
}
Figure