Temporal Consistency Objectives Regularize the Learning of Disentangled Representations

Gabriele Valvano, Agisilaos Chartsias, Andrea Leo, Sotirios A. Tsaftaris

Abstract

There has been an increasing focus in learning interpretable feature representations, particularly in applications such as medical image analysis that require explainability, whilst relying less on annotated data (since annotations can be tedious and costly). Here we build on recent innovations in style-content representations to learn anatomy, imaging characteristics (appearance) and temporal correlations. By introducing a self-supervised objective of predicting future cardiac phases we improve disentanglement. We propose a temporal transformer architecture that given an image conditioned on phase difference, it predicts a future frame. This forces the anatomical decomposition to be consistent with the temporal cardiac contraction in cine MRI and to have semantic meaning with less need for annotations. We demonstrate that using this regularization, we achieve competitive results and improve semi-supervised segmentation, especially when very few labelled data are available. Specifically, we show Dice increase of up to 19% and 7% compared to supervised and semi-supervised approaches respectively on the ACDC dataset.

Introduction

Deep learning has revolutionized medical imaging, yet it remains constrained by a heavy reliance on large, expertly annotated datasets. In cardiac MRI, manually labeling every frame of a beating heart is a labor-intensive, expensive, and error-prone process. To mitigate this, researchers have turned to disentangled representations, which aim to decompose an image into two independent factors: anatomy ($s$), representing the underlying structure (the "what" and "where"), and modality ($z$), representing imaging characteristics like contrast, noise, or brightness (the "how it looks").

While previous frameworks like the Spatial Decomposition Network (SDNet) successfully achieved this separation, they often overlooked the temporal dimension. In cine-MRI sequences, cardiac anatomy follows a predictable physical trajectory. This paper introduces SDTNet, which leverages these temporal dynamics as a self-supervised regularizer to improve disentanglement and segmentation performance, particularly in low-data regimes.

Methodology: The SDTNet Framework

The core innovation of SDTNet is the integration of a Temporal Transformer into the spatial decomposition pipeline. The model decomposes a 2D input image $x_t$ into an anatomy factor $s_t$ and a modality factor $z$.

The SDTNet architecture featuring the Temporal Transformer for self-supervised consistency.

The Temporal Transformer is a modified U-Net architecture designed to predict future anatomical states. Given the current anatomy $s_t$ and a time interval $dt$, it predicts the anatomy at a future time step, $\tilde{s}_{t+dt}$. This process introduces a powerful self-supervision signal: because the model is trained on video sequences, it can compare its predicted future anatomy against the actual anatomy extracted from the subsequent frame. By forcing the anatomy factor to be "warpable" across time, the model is regularized to ensure that $s$ contains pure structural information that obeys the laws of cardiac motion, effectively separating it from static modality characteristics.

Experimental Setup

The researchers evaluated SDTNet using the 2017 Automatic Cardiac Diagnosis Challenge (ACDC) dataset, which consists of 2D cine-MRI images from 100 patients. Manual segmentations of the left ventricle (LV), right ventricle (RV), and myocardium (MYO) served as the ground truth. The dataset was split into 70 patients for training, 15 for validation, and 15 for testing. To test the model's robustness, experiments were conducted in semi-supervised scenarios with varying amounts of labeled data (3%, 6%, 12.5%, 25%, and 100%).

Results and Discussion

The inclusion of temporal consistency as a teacher proved to be a highly effective regularizer. The model demonstrated that temporal dynamics provide a strong signal for understanding cardiac motion without requiring additional human labels.

Qualitative comparison of segmentation results across different labeled data percentages for UNet, SDNet, and SDTNet.

Performance in Low-Data Regimes

The most significant gains were observed when labeled data was extremely scarce. In the "3% Challenge," where only a tiny fraction of the training data was annotated, SDTNet achieved a 63.74% Dice score. This represents a nearly 19% improvement over a standard supervised U-Net (44.89%) and a 7% improvement over the previous state-of-the-art SDNet (56.89%). Even with 100% of labels, SDTNet maintained a competitive edge with an 85.83% Dice score.

Modality Invariance and Synthesis

Because the temporal transformer operates exclusively on the anatomy factor ($s$), its ability to predict future shapes without modality information ($z$) confirms successful disentanglement. Furthermore, the model can synthesize future cardiac phases by combining predicted anatomy with a fixed modality code, effectively "animating" a static heart image.

Conclusion and Future Work

SDTNet demonstrates that temporal consistency is a powerful tool for regularizing disentangled representations in medical imaging. By learning the "rhythm" of the heart, the model achieves superior segmentation performance with minimal supervision.

However, limitations remain. The current transformer learns an "average" motion model and may not yet account for specific pathologies like arrhythmias or localized scarring. Future work could extend this framework from 2D slices to 4D (3D + time) volumes and further validate the clinical utility of the synthesized cardiac sequences.

Code is available at: https://github.com/gvalvano/sdtnet

Summary updated

Abstract

There has been an increasing focus in learning interpretable feature representations, particularly in applications such as medical image analysis that require explainability, whilst relying less on annotated data (since annotations can be tedious and costly). Here we build on recent innovations in style-content representations to learn anatomy, imaging characteristics (appearance) and temporal correlations. By introducing a self-supervised objective of predicting future cardiac phases we improve disentanglement. We propose a temporal transformer architecture that given an image conditioned on phase difference, it predicts a future frame. This forces the anatomical decomposition to be consistent with the temporal cardiac contraction in cine MRI and to have semantic meaning with less need for annotations. We demonstrate that using this regularization, we achieve competitive results and improve semi-supervised segmentation, especially when very few labelled data are available. Specifically, we show Dice increase of up to 19\% and 7\% compared to supervised and semi-supervised approaches respectively on the ACDC dataset. Code is available at: https://github.com/gvalvano/sdtnet .
BibTeX
@inproceedings{valvano2019temporal,
  author = {Gabriele Valvano and Agisilaos Chartsias and Andrea Leo and Sotirios A. Tsaftaris},
  title = {{Temporal Consistency Objectives Regularize the Learning of Disentangled Representations}},
  booktitle = {DART/MIL3ID @ MICCAI},
  year = {2019},
  url = {https://openreview.net/forum?id=TVloUgbo9q},
}
Figure