A Point in the Right Direction: Vector Prediction for Spatially-Aware Self-Supervised Volumetric Representation Learning

Yejia Zhang, Pengfei Gu, Nishchal Sapkota, Hao Zheng, Peixian Liang, Danny Z. Chen

TL;DR

VectorPOSE is a new self-supervised learning framework that teaches AI models "spatial common sense" for 3D medical images. By predicting vectors to anatomical landmarks and focusing on sharp organ boundaries, it significantly improves 3D segmentation performance, especially when labeled data is scarce.

What Problem Does This Paper Address?

In 3D medical imaging (like CT or MRI scans), labeling every voxel is incredibly expensive and time-consuming. To solve this, researchers use Self-Supervised Learning (SSL), where a model "pre-trains" on unlabeled data by solving "pretext tasks" (like a digital jigsaw puzzle) before being fine-tuned for a specific medical task.

However, existing SSL methods often miss the big picture. They might focus too much on local textures (reconstruction) or require massive datasets (contrastive learning). Crucially, they lack spatial awareness. In human anatomy, position, scale, and orientation are highly predictable—if you see a piece of a liver, you know roughly where it is in the body and what should be next to it. Current models don't explicitly learn these "anatomical priors," leading to errors in structure size and boundary delineation.

Methods

The authors introduce VectorPOSE, which uses two novel pretext tasks to bridge this spatial gap:

  1. Vector Prediction (VP): Instead of just classifying where a crop comes from, the model predicts multiple vectors originating from the crop (center and corners) that point toward a "reference landmark" (usually the center of the 3D volume). These vectors are represented in spherical coordinates (distance and angles). This forces the model to understand the crop’s exact position, its orientation, and its scale (extent) relative to the whole body.
  2. Boundary-Focused Reconstruction (BFR): Standard reconstruction tasks often result in blurry edges. BFR fixes this by requiring the model to reconstruct not just the raw voxels, but also the edges of the organs (extracted using a 3D Scharr filter). It also swaps the standard $L_2$ loss for $L_1$ loss, which is less sensitive to outliers and better at preserving sharp transitions.

The architecture uses a 3D ResNet-50 encoder and a lightweight "UNet-like" decoder, making it compatible with standard medical imaging pipelines.

Datasets

The model was evaluated on three diverse 3D medical imaging datasets:

  • CT MMWHS: Whole heart segmentation (7 cardiac structures).
  • CT BCV: Abdominal organ segmentation (13 classes).
  • MRI MSD-Prostate: Prostate segmentation from MRI scans.

The authors tested the model using varying amounts of labeled data (10%, 25%, 50%, and 100%) to demonstrate its efficiency in "low-label" scenarios.

Key Insights and Ideas

  • Anatomy is a Coordinate System: The core "aha" moment is treating 3D space as a continuous vector field. By predicting vectors to a landmark, the model learns a global coordinate system that is much richer than simple "up/down" classification.
  • Continuous > Discrete: Previous "puzzle-solving" tasks used discrete categories (e.g., "is this rotated 90 or 180 degrees?"). VectorPOSE uses continuous space, which prevents the model from taking "shortcuts" and leads to more stable feature learning.
  • Boundaries are Features: In segmentation, the most important information is at the edges. By explicitly forcing the model to reconstruct boundaries during pre-training, it develops a better "semantic" understanding of where one organ ends and another begins.
  • Synergy of Global and Local: The combination of VP (global spatial context) and BFR (local structural detail) creates a more holistic representation than either task could alone.

Key Results

VectorPOSE consistently outperformed state-of-the-art SSL methods like Models Genesis, SAR, and Rubik++ across all datasets.

  • Data Efficiency: The gains were most dramatic when labeled data was limited. On the CT MMWHS dataset with only 10% labels, VectorPOSE achieved a 83.30% Dice score, significantly beating the next best method (SAR at 81.48%) and training from scratch (79.31%).
  • Robustness: It showed superior performance in both CT and MRI modalities, proving that "spatial awareness" is a universal requirement for volumetric medical analysis.
  • Ablation Success: Removing the "corner vectors" or the boundary reconstruction task led to measurable drops in performance, confirming that every part of the "POSE" (Position, Orientation, Scale, Extent) framework contributes to the final result.

Limitations and Future Work

While powerful, the method currently relies on a "reference landmark" which is often just the center of the volume. The authors note that in cases where the patient isn't centered or the scan is partial, this might be less effective.

Future directions include: * Developing unsupervised methods to automatically extract more meaningful anatomical landmarks (like the center of a specific organ) to use as vector targets. * Exploring how these spatial vectors could be used to align different scans of the same patient (registration). * Testing the framework on even more complex tasks, such as multi-modal fusion where spatial alignment is critical.

Summary updated

Abstract

High annotation costs and limited labels for dense 3D medical imaging tasks have recently motivated an assortment of 3D self-supervised pretraining methods that improve transfer learning performance. However, these methods commonly lack spatial awareness despite its centrality in enabling effective 3D image analysis. More specifically, position, scale, and orientation are not only informative but also automatically available when generating image patches for training. Yet, to date, no work has proposed a pretext task that distills all key spatial features. To fulfill this need, we develop a new self-supervised method, VectorPOSE, which promotes better spatial understanding with two novel pretext tasks: Vector Prediction (VP) and Boundary-Focused Reconstruction (BFR). VP focuses on global spatial concepts (i.e., properties of 3D patches) while BFR addresses weaknesses of recent reconstruction methods to learn more effective local representations. We evaluate VectorPOSE on three 3D medical image segmentation tasks, showing that it often outperforms state-of-the-art methods, especially in limited annotation settings.
BibTeX
@inproceedings{zhang2023point,
  author = {Yejia Zhang and Pengfei Gu and Nishchal Sapkota and Hao Zheng and Peixian Liang and Danny Z. Chen},
  title = {{A Point in the Right Direction: Vector Prediction for Spatially-Aware Self-Supervised Volumetric Representation Learning}},
  booktitle = {ISBI},
  year = {2023},
  doi = {10.1109/isbi53787.2023.10230378},
}
Figure