Abstract
Perceptual distances between images, as measured in the space of pre-trained deep features, have outperformed prior low-level, pixel-based metrics on assessing perceptual similarity. While the capabilities of older and less accurate models such as AlexNet and VGG to capture perceptual similarity are well known, modern and more accurate models are less studied. In this paper, we present a large-scale empirical study to assess how well ImageNet classifiers perform on perceptual similarity. First, we observe a inverse correlation between ImageNet accuracy and Perceptual Scores of modern networks such as ResNets, EfficientNets, and Vision Transformers: that is better classifiers achieve worse Perceptual Scores. Then, we examine the ImageNet accuracy/Perceptual Score relationship on varying the depth, width, number of training steps, weight decay, label smoothing, and dropout. Higher accuracy improves Perceptual Score up to a certain point, but we uncover a Pareto frontier between accuracies and Perceptual Score in the mid-to-high accuracy regime. We explore this relationship further using a number of plausible hypotheses such as distortion invariance, spatial frequency sensitivity, and alternative perceptual functions. Interestingly we discover shallow ResNets and ResNets trained for less than 5 epochs only on ImageNet, whose emergent Perceptual Score matches the prior best networks trained directly on supervised human perceptual judgements.
Introduction
In computer vision, the prevailing assumption has been that increasing a model's accuracy on ImageNet leads to internal representations that more closely align with human visual perception. This paper challenges that assumption, revealing a counter-intuitive trade-off: as modern models become more accurate at classification, they often become worse at judging perceptual similarity. This finding has significant implications for researchers using "off-the-shelf" features for tasks like style transfer, image super-resolution, and deblurring.
Methodology
The authors conducted a massive empirical study involving 722 different ImageNet models, including ResNets, EfficientNets, and Vision Transformers (ViTs). They performed extensive 1D sweeps across seven hyperparameters: architecture, scale (depth/width), training steps, weight decay, label smoothing, and dropout.
To measure "Perceptual Score" (PS), the study utilized the 2AFC (Two-Alternative Forced Choice) protocol on the BAPPS (Berkeley-Adobe Perceptual Patch Similarity) dataset. A model is shown a reference patch and two distorted versions; it must "choose" which distorted patch is closer to the original based on the distance between its internal feature maps. Alignment with human majority choices determines the score. Models were trained on 64x64 ImageNet to match the patch size of perceptual benchmarks.
Key Insights
The study identifies a clear "U-shaped" relationship between classification accuracy and perceptual similarity. While improving accuracy helps in the low-accuracy regime, a Pareto frontier emerges in the mid-to-high accuracy regime where further gains in accuracy lead to a decline in Perceptual Score.
Surprisingly, "underfit" or shallower models often outperformed state-of-the-art converged models. For instance, a shallow ResNet-6 or a ResNet-200 trained for fewer than 5 epochs achieved better alignment with human intuition than more accurate counterparts. This phenomenon is architecture-agnostic, affecting CNNs and ViTs alike.
Key Results
The inverse correlation in the high-accuracy regime is strong, with a coefficient of -0.84. Modern architectures like EfficientNets and ViTs consistently show this trade-off, as illustrated below.
Key findings include:
- ResNet-6 achieved a Perceptual Score of 69.1, significantly higher than the more accurate EfficientNet-B5 (67.2) and even beating the classic AlexNet (68.9).
- By early-stopping a ResNet, the authors achieved an emergent Perceptual Score of 70.2, matching the performance of models specifically trained on human perceptual data.
- Dropout was the only hyperparameter found to negatively impact both accuracy and perceptual similarity simultaneously.
Conclusion
This research challenges the "bigger is better" mantra in representation learning. While the exact cause of this trade-off remains unknown—hypotheses regarding spatial frequency sensitivity and feature entropy were ruled out—the practical takeaway is clear: for tasks requiring human-like perceptual judgment, the most accurate ImageNet classifier may not be the best choice. Future work is needed to explore if this trade-off extends to high-level semantic similarity and to identify the underlying factors driving this divergence from human perception.
Summary updated