Perceptual Inductive Bias Is What You Need Before Contrastive Learning

Tianqin Li, Junru Zhao, Dunhan Jiang, Shenghao Wu, Alan Ramirez, Tai Sing Lee

TL;DR

Standard contrastive learning models often suffer from texture bias and slow convergence because they jump straight to high-level semantics, skipping basic visual structures like shapes and boundaries. By introducing a "pre-pretraining" phase inspired by human vision that focuses on shape prototypes and intrinsic images (reflectance and shading), this paper achieves 2x faster convergence and superior performance across classification, segmentation, and depth estimation tasks.

Overview of the proposed pre-pretraining pipeline vs traditional contrastive learning.

What Problem Does This Paper Address?

Self-supervised contrastive learning (CL) has revolutionized computer vision by learning powerful representations without human labels. It works by mapping different augmented views of the same image close together in a latent space. However, CL typically tries to learn high-level object semantics directly from raw pixels.

This skips a crucial step found in human biology. According to David Marr’s seminal theory of perception, human vision is a multi-stage process: our brains map out boundaries, shapes, and surface properties before figuring out what the object actually is. By bypassing this intermediate stage, standard CL models suffer from two major flaws:

  1. Slow Convergence: They take a long time to learn structural regularities from scratch.
  2. Texture Bias: They often exploit local textures (a shortcut) rather than global shapes to identify objects, making them fragile when facing out-of-distribution or stylized data.

Methods

The authors propose a hybrid framework that injects mid-level perceptual inductive biases into the network before running standard contrastive learning. They introduce three distinct approaches:

  1. Shape Prototypical Contrastive Learning (S-PCL): The model uses offline-generated 2D shape silhouettes. These silhouettes are passed through a momentum encoder and clustered using K-Means to discover "shape prototypes" (cluster centroids). A specialized loss function (ShapeProtoNCE) then forces the representation of the original image to align with its corresponding global shape prototype.
  2. Intrinsic Image Decomposition: Using the classic Retinex algorithm, the framework splits an image into two components: reflectance (surface material properties) and shading (3D shape and illumination). These are treated as specialized augmented views within an InfoNCE loss framework.
  3. Mid-level Vision Contrastive Learning (MidVCL): This approach combines both shape prototypes and intrinsic image decompositions into a single unified loss function.
Figure 2. Illustration of Shape Prototypical Contrastive Learn- ing (S-PCL) training framework The original images are input to the encoder and the offline-generated shape silhouettes gener- ated are input to the momentum encoder. The K-Means is per- formed on the embedding of shape silhouettes. Similar shapes are clustered together and the centroid of each cluster is considered to be the shape prototype. The objective function, ShapeProtoNCE, is calculated on the representation of the original image and its corresponding shape prototype.
Figure 3. Illustration of the training framework that incorpo- rates the intrinsic image as a view The original image is input to the encoder and one intrinsic image decomposed by Retinex algo- rithm is input to the momentum encoder. The InfoNCE between the representation of the original image and that of the intrinsic image is computed.

Crucially, the authors discovered that focusing purely on shape causes performance to saturate early because shape alone cannot capture fine-grained semantics. To solve this, they introduce Sequential Training: they warm up the model using S-PCL or MidVCL for the first 100 epochs, and then switch to traditional semantic contrastive learning (like MoCoV2 or PCL) for the remaining epochs.

Datasets

The models were trained and evaluated using several prominent benchmarks:

  • Pretraining: ImageNet-100 (a 100-class subset of ImageNet) and STL-10.
  • Linear Evaluation & Transfer Learning: ImageNet-100 and full ImageNet-1K validation sets.
  • Pixel-level Downstream Tasks: ADE20K and Cityscapes (for semantic segmentation), and NYU Depth V2 (for depth estimation).
  • Robustness & Bias Evaluation: The model-vs-human benchmark, which includes 17 out-of-distribution (OOD) datasets featuring cue conflicts (e.g., an image of a cat overlaid with an elephant's skin texture).

Key Insights and Ideas

  • Shape is an Excellent Starter, but a Poor Finisher: Early in training, clustering by shape aligns well with semantic categories and accelerates feature learning. However, as training progresses, shape-based clustering diverges from true semantic categories. This justifies the sequential "coarse-to-fine" training strategy.
  • Divide and Conquer with Intrinsic Images: Reflectance maps surface details, which heavily benefits image classification and semantic segmentation but fails at depth estimation. Conversely, shading captures 3D contours, significantly boosting depth estimation performance while contributing little to classification.
  • Resolving the Pixel vs. Semantic Trade-off: Standard discriminative models usually excel at either image-level classification or pixel-level segmentation, rarely both. Pretraining with perceptual biases bridges this gap, creating a representation space that serves both high-level and low-level downstream tasks effectively.
  • True Shape Bias Realization: SmoothGrad sensitivity maps reveal that while standard models focus heavily on local textures, S-PCL forces the network to focus squarely on object contours, even when severe style transfers are applied.
Figure 5. A. AMI between cluster assignment of S-PCL trained on ImageNet-100 and ground truth label. The AMI increases at the early stage and decreases later on. B. AMI between cluster assignment of PCL and S-PCl+PCL trained on ImageNet-100 and ground truth label. The AMI of PCL keeps increasing. S-PCL warm-up moves the curve upward. C and D. Linear classification accuracy on two datasets for PCL, S-PCL and S-PCL+PCL. S-PCL suffers rapid saturation but is able to serve as a warm-up strategy to accelerate training.

Key Results

  • 2x Faster Convergence: The hybrid S-PCL + PCL approach achieves classification accuracy comparable to a full 400-epoch PCL baseline in fewer than half the training epochs.
  • State-of-the-Art Downstream Performance: MidVCL achieved the highest performance on pixel-level tasks, scoring 32.1% mIoU on ADE20K and 68.3% mIoU on Cityscapes.
  • Human-like Vision: On the model-vs-human benchmark, S-PCL was the only model evaluated that achieved a shape bias score greater than 0.5, meaning it genuinely prioritized shape over texture for object recognition—closely mimicking human psychophysics.
  • OOD Robustness: S-PCL demonstrated statistically significant improvements in classification accuracy across all 17 out-of-distribution datasets compared to standard contrastive baselines.

Limitations and Future Work

  • Scalability: Due to compute constraints, the authors primarily evaluated their approach using a relatively small architecture (ResNet18) and smaller datasets (ImageNet-100). Verifying how these perceptual biases scale to large Vision Transformers (ViTs) and massive datasets remains open.
  • Narrow Perceptual Scope: The paper focuses strictly on shape silhouettes, reflectance, and shading. Human vision utilizes many other mid-level visual cues—such as surface normals, binocular disparity, and motion parallax—which could be explored in future self-supervised learning frameworks.

Summary updated

Abstract

David Marr's seminal theory of human perception stipulates that visual processing is a multi-stage process, prioritizing the derivation of boundary and surface properties before forming semantic object representations. In contrast, contrastive representation learning frameworks typically bypass this explicit multi-stage approach, defining their objective as the direct learning of a semantic representation space for objects. While effective in general contexts, this approach sacrifices the inductive biases of vision, leading to slower convergence speed and learning shortcut resulting in texture bias. In this work, we demonstrate that leveraging Marr's multi-stage theory-by first constructing boundary and surface-level representations using perceptual constructs from early visual processing stages and subsequently training for object semantics-leads to 2x faster convergence on ResNet18, improved final representations on semantic segmentation, depth estimation, and object recognition, and enhanced robustness and out-of-distribution capability. Together, we propose a pretraining stage before the general contrastive representation pretraining to further enhance the final representation quality and reduce the overall convergence time via inductive bias from human vision systems.
BibTeX
@inproceedings{li2025perceptual,
  author = {Tianqin Li and Junru Zhao and Dunhan Jiang and Shenghao Wu and Alan Ramirez and Tai Sing Lee},
  title = {{Perceptual Inductive Bias Is What You Need Before Contrastive Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00899},
}
Figure