ChronoMID - Cross-Modal Neural Networks for 3-D Temporal Medical Imaging Data

Alexander G. Rakowski, Petar Velickovic, Enrico Dall'Ara, Pietro Liò

Abstract

ChronoMID builds on the success of cross-modal convolutional neural networks (X-CNNs), making the novel application of the technique to medical imaging data. Specifically, this paper presents and compares alternative approaches—timestamps and difference images—to incorporate temporal information for the classification of bone disease in mice, applied to micro-CT scans of mouse tibiae. Whilst much previous work on diseases and disease classification has been based on mathematical models incorporating domain expertise and the explicit encoding of assumptions, the approaches given here utilize the growing availability of computing resources to analyze large datasets and uncover subtle patterns in both space and time. After training on a balanced set of over 75,000 images, all models incorporating temporal features outperformed a state-of-the-art CNN baseline on an unseen, balanced validation set comprising over 20,000 images. The top-performing model achieved 99.54% accuracy, compared to 73.02% for the CNN baseline.

Introduction: The Challenge of Temporal Medical Imaging

In clinical medicine, a single diagnostic snapshot is rarely sufficient. Diseases such as osteoporosis or osteoarthritis develop over months or years, and the efficacy of a treatment can only be judged by how a patient’s condition changes over time. While Recurrent Neural Networks (RNNs) are the standard tool for sequential data, they struggle with the realities of clinical practice, where data is often sparse or irregular. Standard Convolutional Neural Networks (CNNs), while excellent at identifying spatial patterns, remain "blind" to temporal context. ChronoMID addresses this by giving CNNs a sense of history without the rigid requirements of sequential RNNs.

Visualization of 3D micro-CT scan and cross-sectional slices of a mouse tibia, illustrating the spatial complexity of the input data.

Methods: Incorporating Time into CNNs

The researchers developed ChronoMID using a Cross-Modal Convolutional Neural Network (X-CNN) architecture. The core innovation is processing different "modes" of information—the current image and the temporal context—through parallel branches that interact via "X-chains." In these chains, feature maps from the original image and a "difference image" are merged using $1 \times 1$ convolutions, allowing the network to learn which spatial changes are most relevant to the time elapsed.

Two differencing strategies were compared:

  • Absolute: Comparing the current scan ($I_t$) to the initial reference scan ($I_{t0}$).
  • Relative: Comparing the current scan to the one immediately preceding it ($I_{t-1}$).

Additionally, integer timestamps representing the study week were concatenated into the fully connected layers to provide explicit temporal grounding.

Datasets: Longitudinal Bone Remodeling Study

The study utilized a longitudinal dataset of micro-computed tomography (µCT) scans of mouse tibiae. The dataset followed 15 female mice over 8 weeks (ages 14 to 22 weeks), divided into three groups: a healthy "Wild Type" control, a group treated with parathyroid hormone (PTH) to induce bone formation, and an ovariectomy (OVX) group to simulate bone loss. The raw 3D volumes were sliced into over 75,000 training images and 20,000 validation images.

Experimental timeline and group categorization (Wild Type, OVX, PTH) for the longitudinal study of bone disease progression.

Results and Key Insights

The performance gains were dramatic. The top-performing model, utilizing both timestamps and absolute difference images, achieved 99.54% accuracy, compared to just 73.02% for the standard CNN baseline.

Key insights include:

  • Absolute > Relative: Absolute differencing significantly outperformed relative differencing. This is likely because bone remodeling in these models follows a monotonic trend where changes accumulate consistently over time, making the long-term "drift" a stronger signal than noisy week-to-week changes.
  • Training Stability: The temporal models demonstrated superior training stability, reaching over 98% accuracy in as few as 5 epochs, suggesting that temporal features significantly simplify the optimization landscape.

Discussion and Future Work

By treating time as a separate input modality rather than a hidden state, ChronoMID becomes robust to missing data—a common issue in clinical settings. If a scan is missed, the model can still compare the current state to the reference without breaking a sequence.

However, the approach increases computational cost, requiring roughly four times the training time of a standard CNN due to dual-branch complexity and disk I/O. Future work will explore "difference-only" models to reduce memory usage and extend the framework to multi-modal data, such as combining MRI and CT scans.

Summary updated

Abstract

ChronoMID builds on the success of cross-modal convolutional neural networks (X-CNNs), making the novel application of the technique to medical imaging data. Specifically, this paper presents and compares alternative approaches - timestamps and difference images - to incorporate temporal information for the classification of bone disease in mice, applied to micro-CT scans of mouse tibiae. Whilst much previous work on diseases and disease classification has been based on mathematical models incorporating domain expertise and the explicit encoding of assumptions, the approaches given here utilise the growing availability of computing resources to analyse large datasets and uncover subtle patterns in both space and time. After training on a balanced set of over 75000 images, all models incorporating temporal features outperformed a state-of-the-art CNN baseline on an unseen, balanced validation set comprising over 20000 images. The top-performing model achieved 99.54% accuracy, compared to 73.02% for the CNN baseline.
BibTeX
@inproceedings{rakowski2020chronomid,
  author = {Alexander G. Rakowski and Petar Velickovic and Enrico Dall'Ara and Pietro Liò},
  title = {{ChronoMID - Cross-Modal Neural Networks for 3-D Temporal Medical Imaging Data}},
  booktitle = {PLoS},
  year = {2020},
  doi = {10.1371/journal.pone.0228962},
}
Figure