Perceived face similarity reflects inverse-generative and naturalistic discriminative objectives

Wenxuan Guo, Heiko H. Schütt, Kamila Maria Jozwik, Katherine R. Storrs, Nikolaus Kriegeskorte, Tal GolanarXiv 2026 · arXiv

TL;DR

To uncover the computational principles behind human face recognition, this paper pits six deep neural network models with different training objectives against human similarity judgments. By evaluating the models on "controversial stimuli"—face pairs optimized to drive maximum disagreement between networks—the authors demonstrate that human face perception aligns best with models that prioritize high-level invariant structures (via inverse rendering, face identification, or object classification) and are tuned by natural image statistics.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have become highly successful mechanistic hypotheses for how the human brain processes visual information. However, researchers face a "fortunate conundrum": when evaluated on randomly sampled face images, many fundamentally different models (e.g., those trained for face identification versus unsupervised autoencoding) perform equally well and closely approach the human behavioral noise ceiling. This makes it impossible to determine which computational objectives or "visual diets" actually shape human perception.

The core challenge is that in random image samples, low-level image features (like pixels and local textures) are heavily confounded with high-level features (like identity and 3D shape). This paper addresses this limitation by using optimized, controversial stimuli to disentangle these features and expose diagnostic differences between competing computational hypotheses.

Methods

The authors controlled the network architecture by using a standard VGG-16 backbone across all conditions, varying two main factors:

  1. Training Objective: Discriminative face identification (faceID), unsupervised autoencoding (autoenc), inverse rendering (invRend), and domain-general object classification (objCat).
  2. Visual Diet: Synthetic faces generated from a 3D morphable model versus real-world photographs.
Six candidate models varying by training objective and visual diet.

To break the predictive entanglement between these models, the authors implemented controversial stimulus synthesis via a Bayesian optimal experimental design framework. Instead of random sampling, they mathematically optimized pairs of face images to maximize the disagreement between the models' predicted representational geometries.

They then collected face-dissimilarity judgments from 864 human participants using an online multi-arrangement task, where users dragged and dropped face pairs along a vertical axis to rate their relative similarity.

Datasets

  • Training Datasets:
  • VGGFace2: A large-scale dataset of real-world face photographs used for face identification and autoencoding.
  • Basel Face Model (BFM 2019): A 3D morphable model used to generate synthetic face datasets with precise control over identity (shape/texture) and nuisance variables (pose, expression, lighting).
  • ImageNet: A domain-general dataset of natural images used to train the object categorization baseline.
  • Evaluation Stimuli:
  • Frontal BFM: Synthetic faces with a fixed, forward-facing pose.
  • Pose-varied BFM: Synthetic faces systematically varied along the yaw axis to test viewpoint invariance.
  • StyleGAN3: High-resolution, photorealistic synthetic faces used to test model generalization under naturalistic appearance variation (uncontrolled backgrounds and lighting) without celebrity familiarity confounds.

Key Insights and Ideas

  • The Necessity of Controversial Sampling: Under random sampling, different models make highly entangled predictions because face pairs that differ in high-level properties also tend to differ in low-level properties. Controversial sampling successfully forces models into distinct predictive clusters, exposing their unique strengths and flaws.
Multidimensional scaling (MDS) showing how controversial sampling separates model representational geometries compared to random sampling.
  • High-Level Invariance Over Pixel Fidelity: The human visual system strongly prioritizes identity-relevant, transformation-invariant structures. Models with pixel-reconstruction objectives (like standard autoencoders) fail to generalize because they over-index on low-level image statistics that humans naturally discount.
  • The Importance of a Natural Visual Diet: Models trained on real photographs consistently out-predicted their synthetic-trained counterparts, even when tested on synthetic faces. This indicates that a purely synthetic diet omits critical visual dimensions (e.g., realistic textures, complex illumination) that tune human face perception.
  • No Single Model Explains Everything: Model performance interacts heavily with the test domain. Inverse-generative models excel at handling structured 3D variations (like pose changes), whereas discriminative models trained on real photos excel at handling photorealistic diversity.

Key Results

  • Random vs. Controversial Performance: Under random sampling, all six models performed similarly well. Under controversial sampling, clear performance gaps emerged, and all models fell significantly below the human noise ceiling.
  • The Winning Trio: Pooled across all experimental conditions, three models emerged as top performers and were statistically indistinguishable from each other: inverse rendering (invRend-BFM), face identification on real photos (faceID-VGGFace2), and object classification (objCat-ImageNet).
  • Handling Pose Variation: In the pose-varied experiment, the inverse rendering model and the natural-image autoencoder best matched human judgments, confirming that human face similarity perception heavily deemphasizes viewpoint-related nuisance variation.

Limitations and Future Work

  • Fixed Architecture: The study restricted all computational hypotheses to a single feedforward CNN architecture (VGG-16). Future work is needed to see if these insights hold for alternative architectures like Vision Transformers (ViTs).
  • Lack of Recurrent Dynamics: Human ventral-stream processing involves recurrent feedback loops that handle tasks like iterative shape refinement or switching from face detection to identification. Purely feedforward networks cannot capture these temporal dynamics.
  • Out-of-Distribution Testing: Models were evaluated based on their task-learned representations without any explicit linear or non-linear fitting to human behavioral data. While this strictly tests zero-shot generalization, explicitly fitting the models might yield closer alignment with human perception.

Summary updated

Abstract

The perceptual representations supporting our ability to recognize faces remain a computational mystery. Deep neural networks offer mechanistic hypotheses for human face perception, but theoretically distinct models often make indistinguishable representational predictions for randomly sampled faces. To expose diagnostic differences among these hypotheses, we compared six neural network models sharing an architecture but trained on distinct tasks, using face pairs optimized to elicit contrasting model predictions ("controversial" pairs) alongside randomly sampled pairs. We tested model predictions against face-dissimilarity judgments from 864 human participants across stimulus sets differing in realism and pose variation. Models prioritizing high-level, invariant structures (trained via inverse rendering, face identification, or object classification) most robustly matched human judgments. Furthermore, models trained on natural images typically outperformed synthetic-trained counterparts. Together, these findings suggest that human face perception is shaped by mechanisms that infer latent causes of facial appearance, discount nuisance variation, and are tuned by natural image statistics.
BibTeX
@misc{guo2026perceived,
  author = {Wenxuan Guo and Heiko H. Schütt and Kamila Maria Jozwik and Katherine R. Storrs and Nikolaus Kriegeskorte and Tal Golan},
  title = {{Perceived face similarity reflects inverse-generative and naturalistic discriminative objectives}},
  year = {2026},
  eprint = {2605.12619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.12619},
}

Loading related content...

Figure