Abstract
Isolating different types of motion in video data is a highly relevant problem in video analysis. Applications can be found, for example, in dynamic medical or biological imaging, where the analysis and further processing of the dynamics of interest is often complicated by additional, unwanted dynamics, such as motion of the measurement subject. In this work, it is empirically shown that a representation of video data via untrained generator networks, together with a specific technique for latent space disentanglement that uses minimal, one-dimensional information on some of the underlying dynamics, allows to efficiently isolate different, highly non-linear motion types. In particular, such a representation allows to freeze any selection of motion types, and to obtain accurate independent representations of other dynamics of interest. Obtaining such a representation does not require any pre-training on a training data set, i.e., all parameters of the generator network are learned directly from a single video.
Introduction
In dynamic medical imaging, such as cardiac MRI, researchers often face the challenge of overlapping motions. A video of a beating heart is typically confounded by the patient's respiratory motion, which shifts the entire chest cavity. Separating these dynamics is essential for precise diagnosis and analysis. Traditional linear methods like Independent Component Analysis (ICA) often fail to capture the complex, non-linear deformations inherent in biological tissues. Conversely, modern deep learning models usually require vast amounts of labeled training data, which is often unavailable or difficult to curate for specific clinical cases.
This paper introduces a novel framework that utilizes untrained generator networks to achieve motion separation without any prior training. By leveraging the internal prior of a convolutional neural network (CNN), the method can disentangle complex motions using only a single video sequence and a minimal auxiliary signal.
Methodology: Untrained Generators and Latent Disentanglement
The core innovation lies in the use of an untrained generator network, inspired by the "Deep Image Prior" concept. Instead of learning from a population, the network's parameters are optimized to represent the specific structures and dynamics of a single target video.
Latent Space Partitioning
To achieve motion separation, the authors partition the network's latent space into three distinct components:
- Static Component: Captures the time-invariant anatomical structures.
- Known Dynamic Component: This part is driven by a "trigger" signal—a simple 1D scalar representing a known motion, such as a respiratory curve obtained from a chest strap.
- Unknown Dynamic Component: The network is tasked with finding the remaining motion (e.g., the heartbeat) required to accurately reconstruct the video frames.
By optimizing the network with these constraints, the model learns to map specific motions to specific latent variables. Post-optimization, researchers can "freeze" the respiratory variable at a constant value, generating a synthetic video that shows the heart beating in isolation, effectively simulating a perfect breath-hold.
Experimental Setup and Datasets
The efficacy of the proposed method was evaluated on two primary datasets:
- Synthetic Phantom: A controlled environment consisting of two nested disks. One disk undergoes dilation (simulating cardiac contraction), while the entire scene is subjected to shearing (simulating respiratory motion). This setup allows for precise error measurement against a known ground truth.
- Semi-synthetic Cardiac MRI: Real cine-MRI scans (four-chamber and short-axis views) were augmented with synthetic shearing motion. This tested the model's ability to handle realistic anatomical textures and complex motion patterns.
Results and Discussion
The experimental results demonstrate several key advantages of the untrained generator approach:
- Non-Linear Robustness: Unlike ICA, the CNN-based generator naturally handles non-linear deformations like shearing and stretching. In synthetic tests, the proposed method achieved significantly lower error rates and cleaner "difference images" than linear baselines.
- Zero-Shot Learning: The method operates without pre-training, eliminating the risk of "hallucinations" or biases derived from external datasets. It relies solely on the physics and structures present in the individual patient's scan.
- Minimal Supervision: The reliance on a simple 1D trigger signal is highly practical, as such signals (ECG or respiratory bellows) are often already collected during clinical MRI sessions. The method also showed robustness to noise in these trigger signals.
- Computational Efficiency: Optimization for complex MRI sequences took approximately 3.8 minutes on a consumer-grade GPU (Nvidia RTX 3090), making it feasible for research and potentially clinical workflows.
Conclusion and Future Work
This work presents a powerful, data-efficient solution for motion separation in dynamic imaging. By combining untrained generator networks with structured latent space disentanglement, the authors successfully isolated cardiac and respiratory cycles with high fidelity.
Future research will focus on addressing the non-convexity of the optimization process to ensure more stable results across different initializations. Additionally, the authors aim to integrate this motion separation directly into the image reconstruction pipeline, potentially allowing for high-quality imaging from highly undersampled or noisy raw data.
Summary updated