Vector Contrastive Learning For Pixel-Wise Pretraining In Medical Vision

Yuting He, Shuo Li

TL;DR

This paper introduces "Vector Contrastive Learning" (Vector CL), a novel self-supervised pretraining (SSP) approach for medical vision models. By reformulating contrastive learning as a vector regression problem, Vector CL effectively addresses the "over-dispersion" issue in pixel-wise representations, leading to more robust and generalizable medical visual foundation models.

What Problem Does This Paper Address?

Self-supervised pretraining (SSP) has revolutionized foundation models, but applying it effectively to pixel-wise representations in medical imaging remains a challenge. Standard contrastive learning (CL) methods, designed for global representations, often suffer from an "over-dispersion" problem when adapted to pixel-level tasks. This means that while features are pushed apart, their inherent correlations within a pixel's neighborhood are lost, leading to blurred or indistinguishable fine details crucial for medical diagnosis. Existing attempts to mitigate this issue still rely on the flawed binary CL framework. This paper aims to bridge this gap by proposing a truly pixel-wise CL approach that preserves these vital pixel correlations.

Methods

The core innovation is Vector Contrastive Learning (Vector CL), which reframes CL from a binary optimization problem to a vector regression problem. Instead of directly minimizing or maximizing feature distances, Vector CL models feature distances by regressing displacement vectors in image space. This approach allows for a more nuanced quantification of feature dispersion while preserving pixel-wise correlations.

The proposed COntrast in VEctor Regression (COVER) framework implements this paradigm through three key components:

  1. Self-Vector Regression (SeVR): This component enables an extendable, annotation-free self-supervised learning process. It generates a dense displacement vector field (DVF) by applying random spatial transformations to an image. This DVF acts as the ground truth for training the network to predict pixel-wise correspondences.
  2. Mixture of Vectors (MoV): This module formulates the mapping from feature distances to vectors. It comprises:
  3. Vector Embedding Unit (VEU): Encodes spatial continuity by modeling relationships within a receptive field, preventing artificial divisions and preserving feature correlations.
  4. Multi-Vector Integration (MVI): Addresses correspondence ambiguity by extracting multiple vectors, allowing for adaptability to diverse feature concerns and improving bias adaptability.
  5. Vector Pyramid Aggregation (VPA): This component introduces multiscalarity to correspondence modeling. By stacking MoV modules in a pyramid-like architecture, VPA efficiently captures correspondences across different feature scales, leading to granularity-adapted representations.

Datasets

The study evaluated COVER on a diverse set of medical imaging tasks and modalities:

  • 2D Evaluation: ChestX-ray8 dataset (112,120 chest X-ray images) for tasks like SCR25% (X-ray segmentation), PDCXRC (X-ray classification), FIVESs (fundus vessel segmentation), and KiPA222D (CT segmentation).
  • 3D Evaluation: PPMI database (837 3D T1 brain MR images) for tasks like CANDIS (MR segmentation), FeTA21s (MR segmentation), STOICc (CT classification), and KiPA223D (CT segmentation).

Key Insights and Ideas

  • Overcoming Over-Dispersion: The core insight is that pixel-wise features in medical images are inherently correlated. Standard CL's "over-dispersion" breaks these correlations, hindering the learning of subtle semantic details. Vector CL directly tackles this by modeling distances as vectors, preserving these crucial relationships.
  • Vector Regression as a Solution: Reframing CL as vector regression allows for a more principled way to guide distance modeling. By learning to predict displacement vectors, the model implicitly learns feature distances and their spatial relationships.
  • Annotation-Free Learning for Medical Images: SeVR's ability to generate ground truth displacement vectors from random transformations is a significant step towards truly unsupervised learning for medical imaging, reducing reliance on costly annotations.
  • Multiscalarity is Key: VPA's pyramid architecture effectively captures correspondences at various scales, crucial for segmenting both small structures (like vessels) and larger anatomical regions.
  • Adaptability Across Scenes: The method demonstrates strong adaptability to both consistent (e.g., chest X-rays) and inconsistent (e.g., fundus images) medical imaging scenarios, outperforming methods that struggle with domain shifts.

Key Results

COVER significantly outperforms 17 existing methods across 8 downstream tasks in both 2D and 3D medical vision evaluations.

  • Overall Performance: COVER achieved the highest average scores, demonstrating its strong generalizability. For instance, on the SCR25% task, it achieved 94.0% DSC, an improvement of over 5% compared to "Scratch."
  • Cross-Scale Transferability: The framework excels at learning representations for both small structures (e.g., vessels in FIVESs, achieving 87.2% DSC) and large structures (e.g., chest regions in SCR25%, achieving 94.0% DSC).
  • Cross-Scene Adaptability: COVER shows robust performance across diverse medical imaging scenes, improving by over 5% compared to "Scratch" on average. It also mitigates the performance degradation seen in other methods (like DenseCL and VADER) on inconsistent scenes.
  • Efficiency: COVER requires significantly fewer fine-tuned parameters and less fine-tuning data compared to other methods, making it more practical. For example, it achieved DSC close to 90% using only 5% of the training data.
  • Convergence: The framework exhibits superior convergence ability during fine-tuning and enhanced discrimination during pretraining, especially when the vector regression loss is incorporated.

Limitations and Future Work

  • Dataset Scale: The authors suggest that involving larger-scale medical image datasets in training is a crucial next step to foster the development of truly large pixel-wise foundation models in the medical field.
  • Hyperparameter Sensitivity: While the paper analyzes the impact of hyperparameters like receptive field size (N) and the number of vector groups (J) in MoV, further exploration might be beneficial for optimal performance across all scenarios.
  • Generalization to Continuous Signals: The paper highlights that Vector CL opens new frontiers in representations for continuous signals, implying that further research could explore its application beyond traditional medical imaging tasks.

Summary updated

Abstract

Contrastive learning (CL) has become a cornerstone of self-supervised pretraining (SSP) in foundation models, however, extending CL to pixel-wise representation, crucial for medical vision, remains an open problem. Standard CL formulates SSP as a binary optimization problem (binary CL) where the excessive pursuit of feature dispersion leads to an over-dispersion problem, breaking pixel-wise feature correlation thus disrupting the intra-class distribution. Our vector CL reformulates CL as a vector regression problem, enabling dispersion quantification in pixel-wise pretraining via modeling feature distances in regressing displacement vectors. To implement this novel paradigm, we propose the COntrast in VEctor Regression (COVER) framework. COVER establishes an extendable vector-based self-learning, enforces a consistent optimization flow from vector regression to distance modeling, and leverages a vector pyramid architecture for granularity adaptation, thus preserving pixel-wise feature correlations in SSP. Extensive experiments across 8 tasks, spanning 2 dimensions and 4 modalities, show that COVER significantly improves pixel-wise SSP, advancing generalizable medical visual foundation models.
BibTeX
@inproceedings{he2025vector,
  author = {Yuting He and Shuo Li},
  title = {{Vector Contrastive Learning For Pixel-Wise Pretraining In Medical Vision}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure