Do Computer Vision Foundation Models Learn the Low-level Characteristics of the Human Visual System?

Yancheng Cai, Fei Yin, Dounia Hammou, Rafal MantiukCVPR 2025 · DOI

Introduction

Modern computer vision foundation models (FMs) like DINOv2 and CLIP are trained on vast datasets of natural images—the same visual statistics that have shaped human evolution. This raises a fundamental question: do these models converge on the same low-level visual solutions as the human visual system (HVS)? This paper investigates whether AI models share human-like "bottlenecks," such as specific spatial frequency sensitivities and contrast masking effects.

Conceptual framework comparing the Human Visual System and Image Encoders across contrast detection, masking, and constancy tasks.

Methodology

To evaluate these models, the researchers treated neural networks as human subjects in a psychophysics lab. They utilized synthetic stimuli, primarily Gabor patches and band-limited noise, to test three core characteristics:

  1. Contrast Detection: The minimum signal required to perceive a pattern against a flat background.
  2. Contrast Masking: The difficulty of perceiving a pattern when obscured by a "masking" background.
  3. Contrast Constancy: The perceived intensity of high-contrast patterns across different scales.

The study evaluated 45 models (including DINO, DINOv2, OpenCLIP, and SAM) by measuring the cosine similarity ($S_{ac}$) between feature vectors of test and reference images. These responses were then compared against the castleCSF gold-standard human vision model.

Key Insights

The study revealed several critical points of alignment and divergence:

  • Convergence in Masking: The strongest agreement between AI and humans occurred in contrast masking. This suggests that the ability to distinguish signal from background noise is a universal requirement for any effective vision system.
  • DINOv2's Human-like Vision: DINOv2 emerged as the most human-aligned model. Its self-supervised training objective appears to capture fundamental aspects of visual representation more effectively than other methods.
  • The Sensitivity Gap: AI models generally exhibit significantly lower sensitivity to low-contrast patterns compared to humans, often failing to detect details that are easily visible to the human eye.

Quantitative Results

The researchers found that while some models like DINOv2 and SD-VAE show hints of the human-like "peak" in spatial frequency sensitivity, many others, such as OpenCLIP, display jagged and inconsistent responses.

Spearman correlation and RMSE results across 45 models for various contrast detection and masking tasks, highlighting DINOv2's superior alignment with human data.

Quantitatively, DINOv2 variants consistently achieved the highest Spearman correlations with human psychophysical data across nine different test types, as shown in the performance summary above. In contrast, models like OpenCLIP showed high inconsistency across different spatial frequencies.

Limitations and Future Directions

A primary limitation identified is the "sRGB bottleneck." Most models are trained on 8-bit web images, which lack the full luminance range processed by the human eye, likely hindering the learning of true low-light sensitivity. Furthermore, it remains unclear whether the success of models like DINOv2 is a result of the Vision Transformer (ViT) architecture or the specific training recipe. Future research could explore how task-specific fine-tuning affects these low-level visual characteristics.

By establishing this bridge between psychophysics and deep learning, the study provides a robust benchmark for measuring the foundational "understanding" of visual building blocks in AI.

Summary updated

BibTeX
@inproceedings{cai2025do,
  author = {Yancheng Cai and Fei Yin and Dounia Hammou and Rafal Mantiuk},
  title = {{Do Computer Vision Foundation Models Learn the Low-level Characteristics of the Human Visual System?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01866},
}
Figure