TL;DR
Standard contrastive learning models often suffer from texture bias and slow convergence because they jump straight to high-level semantics, skipping basic visual structures like shapes and boundaries. By introducing a "pre-pretraining" phase inspired by human vision that focuses on shape prototypes and intrinsic images (reflectance and shading), this paper achieves 2x faster convergence and superior performance across classification, segmentation, and depth estimation tasks.
What Problem Does This Paper Address?
Self-supervised contrastive learning (CL) has revolutionized computer vision by learning powerful representations without human labels. It works by mapping different augmented views of the same image close together in a latent space. However, CL typically tries to learn high-level object semantics directly from raw pixels.
This skips a crucial step found in human biology. According to David Marr’s seminal theory of perception, human vision is a multi-stage process: our brains map out boundaries, shapes, and surface properties before figuring out what the object actually is. By bypassing this intermediate stage, standard CL models suffer from two major flaws:
- Slow Convergence: They take a long time to learn structural regularities from scratch.
- Texture Bias: They often exploit local textures (a shortcut) rather than global shapes to identify objects, making them fragile when facing out-of-distribution or stylized data.
Methods
The authors propose a hybrid framework that injects mid-level perceptual inductive biases into the network before running standard contrastive learning. They introduce three distinct approaches:
- Shape Prototypical Contrastive Learning (S-PCL): The model uses offline-generated 2D shape silhouettes. These silhouettes are passed through a momentum encoder and clustered using K-Means to discover "shape prototypes" (cluster centroids). A specialized loss function (
ShapeProtoNCE) then forces the representation of the original image to align with its corresponding global shape prototype. - Intrinsic Image Decomposition: Using the classic Retinex algorithm, the framework splits an image into two components: reflectance (surface material properties) and shading (3D shape and illumination). These are treated as specialized augmented views within an InfoNCE loss framework.
- Mid-level Vision Contrastive Learning (MidVCL): This approach combines both shape prototypes and intrinsic image decompositions into a single unified loss function.
Crucially, the authors discovered that focusing purely on shape causes performance to saturate early because shape alone cannot capture fine-grained semantics. To solve this, they introduce Sequential Training: they warm up the model using S-PCL or MidVCL for the first 100 epochs, and then switch to traditional semantic contrastive learning (like MoCoV2 or PCL) for the remaining epochs.
Datasets
The models were trained and evaluated using several prominent benchmarks:
- Pretraining: ImageNet-100 (a 100-class subset of ImageNet) and STL-10.
- Linear Evaluation & Transfer Learning: ImageNet-100 and full ImageNet-1K validation sets.
- Pixel-level Downstream Tasks: ADE20K and Cityscapes (for semantic segmentation), and NYU Depth V2 (for depth estimation).
- Robustness & Bias Evaluation: The model-vs-human benchmark, which includes 17 out-of-distribution (OOD) datasets featuring cue conflicts (e.g., an image of a cat overlaid with an elephant's skin texture).
Key Insights and Ideas
- Shape is an Excellent Starter, but a Poor Finisher: Early in training, clustering by shape aligns well with semantic categories and accelerates feature learning. However, as training progresses, shape-based clustering diverges from true semantic categories. This justifies the sequential "coarse-to-fine" training strategy.
- Divide and Conquer with Intrinsic Images: Reflectance maps surface details, which heavily benefits image classification and semantic segmentation but fails at depth estimation. Conversely, shading captures 3D contours, significantly boosting depth estimation performance while contributing little to classification.
- Resolving the Pixel vs. Semantic Trade-off: Standard discriminative models usually excel at either image-level classification or pixel-level segmentation, rarely both. Pretraining with perceptual biases bridges this gap, creating a representation space that serves both high-level and low-level downstream tasks effectively.
- True Shape Bias Realization: SmoothGrad sensitivity maps reveal that while standard models focus heavily on local textures, S-PCL forces the network to focus squarely on object contours, even when severe style transfers are applied.
Key Results
- 2x Faster Convergence: The hybrid
S-PCL + PCLapproach achieves classification accuracy comparable to a full 400-epoch PCL baseline in fewer than half the training epochs. - State-of-the-Art Downstream Performance: MidVCL achieved the highest performance on pixel-level tasks, scoring 32.1% mIoU on ADE20K and 68.3% mIoU on Cityscapes.
- Human-like Vision: On the model-vs-human benchmark, S-PCL was the only model evaluated that achieved a shape bias score greater than 0.5, meaning it genuinely prioritized shape over texture for object recognition—closely mimicking human psychophysics.
- OOD Robustness: S-PCL demonstrated statistically significant improvements in classification accuracy across all 17 out-of-distribution datasets compared to standard contrastive baselines.
Limitations and Future Work
- Scalability: Due to compute constraints, the authors primarily evaluated their approach using a relatively small architecture (ResNet18) and smaller datasets (ImageNet-100). Verifying how these perceptual biases scale to large Vision Transformers (ViTs) and massive datasets remains open.
- Narrow Perceptual Scope: The paper focuses strictly on shape silhouettes, reflectance, and shading. Human vision utilizes many other mid-level visual cues—such as surface normals, binocular disparity, and motion parallax—which could be explored in future self-supervised learning frameworks.
Summary updated