Abstract
Contrastive learning (CL) has become a cornerstone of self-supervised pretraining (SSP) in foundation models; however, extending CL to pixel-wise representation, crucial for medical vision, remains an open problem. Standard CL formulates SSP as a binary optimization problem (binary CL) where the excessive pursuit of feature dispersion leads to an over-dispersion problem, breaking pixel-wise feature correlation and thus disrupting the intra-class distribution. Our vector CL reformulates CL as a vector regression problem, enabling dispersion quantification in pixel-wise pretraining via modeling feature distances in regressing displacement vectors. To implement this novel paradigm, we propose the COntrast in VEctor Regression (COVER) framework. COVER establishes an extendable vector-based self-learning, enforces a consistent optimization flow from vector regression to distance modeling, and leverages a vector pyramid architecture for granularity adaptation, thus preserving pixel-wise feature correlations in SSP. Extensive experiments across 8 tasks, spanning 2 dimensions and 4 modalities, show that COVER significantly improves pixel-wise SSP, advancing generalizable medical visual foundation models.
The Over-Dispersion Problem in Pixel-Wise Learning
Contrastive Learning (CL) is the current "gold standard" for self-supervised pretraining. Typically, a model learns by pulling "positive" pairs (different views of the same image) together and pushing "negative" pairs (different images) apart. While effective for global image classification, this approach falters in pixel-wise tasks, such as segmenting fine blood vessels or lesions.
In medical imaging, semantic information often changes continuously (e.g., the boundary of a tumor). Standard binary CL incentivizes the model to maximize the distance between all different features. This leads to "over-dispersion", where adjacent pixels are pushed too far apart in the embedding space. This disruption of natural spatial correlations makes the model "blind" to subtle variations and fine-grained structures, often resulting in blurred or misidentified features.
The COVER Framework: Vector Contrastive Learning
To address these limitations, we introduce the COVER (COntrast in VEctor Regression) framework. COVER shifts the paradigm from binary similarity matching to Vector Contrastive Learning. Instead of asking "is this the same?", the model predicts "how far and in what direction" pixels have moved following a spatial transformation.
The framework consists of three core innovations:
- Self-Vector Regression (SeVR): The system applies random spatial transformations (rotation, scaling) to an image, creating a "Displacement Vector Field" (DVF) that serves as the ground-truth map for the model to predict.
- Mixture of Vectors (MoV): This module maps feature distances to vectors using a "vector template matrix" to ensure spatial continuity. It handles "correspondence ambiguity"—where one pixel might resemble several others—by integrating multiple potential vectors into a single robust prediction.
- Vector Pyramid Aggregation (VPA): To handle features at multiple scales, VPA uses a pyramid architecture. Lower levels manage global, coarse-grained correspondences (e.g., organ position), while higher levels focus on fine-grained, local details (e.g., specific vessel branches).
Experimental Evaluation
COVER was evaluated across 8 downstream tasks involving 4 imaging modalities (X-ray, MRI, CT, and Fundus images). Pretraining was conducted on large-scale datasets like ChestX-ray8 (2D) and PPMI (3D).
Key Results
- Superior Performance: COVER consistently outperformed 17 state-of-the-art methods, including SimCLR, MoCo, and BYOL.
- Fine Structure Mastery: Significant gains were observed in complex structures, with a +7.8% improvement in Dice Similarity Coefficient (DSC) for retinal vessel segmentation and +6.7% for fetal brain tissue.
- Data Efficiency: COVER demonstrated remarkable robustness; fine-tuning with only 5% of training data yielded results comparable to other methods using 100% of the data.
- Computational Efficiency: The VPA structure achieved 52x better computational efficiency than direct large-template approaches, allowing for a large receptive field without the overhead of standard attention mechanisms.
Conclusion
By reformulating contrastive learning as a vector regression problem, COVER effectively mitigates the over-dispersion problem inherent in pixel-wise pretraining. This approach preserves essential spatial correlations, enabling the development of more accurate and generalizable medical visual foundation models. Future work will focus on scaling COVER to even larger datasets and more diverse medical modalities.
Summary updated