Using the Order of Tomographic Slices as a Prior for Neural Networks Pre-Training

Yaroslav Zharov, Alexey Ershov, Tilo Baumbach, Vincent HeuvelinearXiv 2022 · arXiv · OpenReview

TL;DR

This paper introduces "SortingLoss," a novel self-supervised pre-training method for neural networks that leverages the inherent structural order of tomographic (CT) scan slices. By training models to predict the correct sequence of slices from a volume, SortingLoss enables more efficient fine-tuning on sparse annotations, significantly reducing annotation effort and computational costs.

What Problem Does This Paper Address?

Acquiring and annotating large 3D datasets from techniques like Computed Tomography (CT) is a major bottleneck in medical diagnostics and scientific research. Manual annotation is time-consuming, error-prone, and often requires expert knowledge. Current methods typically involve annotating a few 2D slices and then interpolating the rest, which can introduce inaccuracies. While self-supervised learning (SSL) offers a way to learn from unlabeled data, existing SSL methods are often designed for 2D photographic images and don't fully exploit the unique properties of volumetric tomographic data. This paper aims to bridge this gap by developing an SSL method specifically for CT data that reduces the reliance on extensive manual annotation and interpolation.

Methods

The core of the proposed method, "SortingLoss," is a novel pretext task designed for tomographic data. Instead of relying on general-purpose augmentations or complex proxy tasks, SortingLoss assumes that the structural information within a sequence of slices along a specific axis of a CT volume provides enough clues to determine their correct order. The method trains a neural network to predict this order.

Key aspects of the method include:

  • Task-Specific Prior: It leverages a "mild assumption" that the anatomical structure within slices along an axis (e.g., cranio-caudal for a human body) can reveal their correct ordering.
  • No Negative Sampling: Unlike many contrastive SSL methods (like SimCLR), SortingLoss does not require sampling negative pairs, simplifying the training process and reducing computational overhead.
  • Slice-Based Pre-training: The pre-training is performed on individual slices, allowing for efficient fine-tuning on sparse sets of annotated slices without the need for interpolation.
  • Margin Ranking Loss: A margin ranking loss is used to enforce that correctly ordered slice pairs have predicted orders that are sufficiently distinct from incorrect ones.
  • Mean Displacement Metric: An interpretable metric, "Mean Displacement," is proposed to track the progress of the pre-training, measuring how much slices, on average, would need to be shifted to achieve perfect ordering.

Datasets

The authors evaluated their method on two distinct datasets:

  • MosMedData: A dataset containing 1110 volumes of human chest CT scans, with 50 volumes specifically annotated for lung tissues affected by COVID-19.
  • Medaka Visual System Dataset: A large-scale dataset of high-resolution synchrotron-based full-body CT scans of Medaka fish (Oryzias latipes), featuring 21 segmentation annotations for structures within the visual system. This dataset was collected as part of an ongoing project and is planned to be released.

Key Insights and Ideas

  • Exploiting Volumetric Structure: The paper demonstrates that the inherent 3D structure of tomographic data, specifically the sequential ordering of slices, can be a powerful signal for self-supervised learning, even without explicit 3D supervision.
  • Efficiency Gains: By eliminating the need for negative sampling and focusing on a task-specific prior, SortingLoss achieves significant improvements in training speed and memory usage compared to general-purpose SSL methods.
  • Reduced Annotation Burden: The ability to fine-tune effectively on sparse slice annotations directly addresses the critical problem of expensive and time-consuming manual annotation for 3D medical data.
  • Robustness to Augmentations: The method shows robustness and even improved performance when subjected to tougher data augmentations, suggesting its learned features are more resilient.
  • Unsupervised Localization: The learned features can be used to localize samples within volumes in an unsupervised manner, which is a valuable capability for data exploration.

Key Results

  • Performance Parity: SortingLoss performs on par with the popular SimCLR baseline on the MosMedData dataset.
  • Speed and Memory Efficiency: SortingLoss is 2x faster and requires 1.5x less memory than SimCLR. On a specific GPU, it uses 40% less memory and trains twice as fast.
  • Outperformance on Sparse Data: On the Medaka dataset, SortingLoss consistently outperforms SimCLR when fine-tuning on very limited annotated slices (1%, 10%).
  • Improved Fine-tuning: Even when fine-tuning on full data, models pre-trained with SortingLoss showed a 2.3% improvement in IoU compared to training from scratch.
  • Qualitative Improvements: Visual comparisons (Figure 1) suggest that pre-training with SortingLoss leads to more accurate segmentation of smaller structures.
  • Expert Evaluation: While not statistically significant, expert evaluations indicated a preference for segmentations from models pre-trained with SortingLoss over training from scratch.

Limitations and Future Work

  • Assumption on Data Structure: The primary limitation is the assumption that the sample's structure provides a strong clue for ordering along at least one axis. This might not hold for all types of tomographic data.
  • Future Work:
    • Investigating the benefits of other ranking losses that can utilize list-wise ordering information.
  • Exploring the application of SortingLoss to datasets from other tomographic imaging methods like MRI, PET, or neutron tomography.
  • Further research into the unsupervised localization capabilities.
  • The authors plan to release the Medaka visual system dataset after the project concludes.

Summary updated

Abstract

The technical advances in Computed Tomography (CT) allow to obtain immense amounts of 3D data. For such datasets it is very costly and time-consuming to obtain the accurate 3D segmentation markup to train neural networks. The annotation is typically done for a limited number of 2D slices, followed by an interpolation. In this work, we propose a pre-training method SortingLoss. It performs pre-training on slices instead of volumes, so that a model could be fine-tuned on a sparse set of slices, without the interpolation step. Unlike general methods (e.g. SimCLR or Barlow Twins), the task specific methods (e.g. Transferable Visual Words) trade broad applicability for quality benefits by imposing stronger assumptions on the input data. We propose a relatively mild assumption -- if we take several slices along some axis of a volume, structure of the sample presented on those slices, should give a strong clue to reconstruct the correct order of those slices along the axis. Many biomedical datasets fulfill this requirement due to the specific anatomy of a sample and pre-defined alignment of the imaging setup. We examine the proposed method on two datasets: medical CT of lungs affected by COVID-19 disease, and high-resolution synchrotron-based full-body CT of model organisms (Medaka fish). We show that the proposed method performs on par with SimCLR, while working 2x faster and requiring 1.5x less memory. In addition, we present the benefits in terms of practical scenarios, especially the applicability to the pre-training of large models and the ability to localize samples within volumes in an unsupervised setup.
BibTeX
@misc{zharov2022using,
  author = {Yaroslav Zharov and Alexey Ershov and Tilo Baumbach and Vincent Heuveline},
  title = {{Using the Order of Tomographic Slices as a Prior for Neural Networks Pre-Training}},
  year = {2022},
  eprint = {2203.09372},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.09372},
}
Figure