Face dissimilarity judgements are predicted by representational distance in morphable and image-computable models

Kamila M. Jozwik, Jonathan O’Keeffe, Katherine R. Storrs, Wenxuan Guo, Tal Golan, Nikolaus KriegeskortebioRxiv 2021 · DOI

TL;DR

This paper investigates how well computational models can predict human judgments of face similarity and identity. They found that simple Euclidean distances within a 3D morphable face model (BFM) are surprisingly good predictors, competitive with complex deep neural networks. This suggests that human face perception is tuned to the statistical distribution of faces in a "face space."

What Problem Does This Paper Address?

Recognizing faces is fundamental to social interaction, yet we lack a precise, quantitative way to predict how similar two faces will appear to humans. While 3D morphable models (like the Basel Face Model, BFM) are excellent for generating realistic faces and capturing their underlying geometric and textural variations, it's unclear how well these models' "face spaces" align with human perceptual judgments of similarity and identity. This research aims to bridge this gap by systematically comparing various computational models, including BFM-based representations and deep neural networks (DNNs), against human behavioral data.

Methods

The authors designed a novel behavioral task where participants arranged pairs of generated faces along a continuum from "same" to "different" identity, and also rated their dissimilarity. They used the Basel Face Model (BFM) to generate 232 unique face pairs, carefully sampling geometric relationships within its principal component space. This controlled stimulus set allowed them to test a wide array of models, including:

  • BFM-based models: Euclidean and cosine distances within the full BFM space, as well as subspaces focusing on shape, texture, or "person attributes" (height, weight, age, gender).
  • Image-computable models: Low-level features like pixels and GIST descriptors, and deep neural networks (DNNs) like VGG and AlexNet trained on various datasets (objects, real faces, or BFM-generated faces).
  • Other models: Active Appearance Model (AAM), Eigenfaces, and configural models based on facial landmarks.

Model performance was evaluated using Representational Similarity Analysis (RSA), correlating model-predicted dissimilarities with human judgments. They also explored the effect of fitting a sigmoidal transform to model predictions and tested a weighted combination of all models.

Datasets

The primary dataset was collected by the authors through their novel behavioral task, involving 26 participants over three sessions. This dataset consists of dissimilarity and identity judgments for 232 pairs of realistic faces generated by the Basel Face Model (BFM). The BFM itself, trained on 200 adult face scans, serves as the basis for generating stimuli and for some of the computational models.

Key Insights and Ideas

  • BFM Euclidean distance is a strong contender: Surprisingly, simple Euclidean distances within the BFM's principal component space explained human dissimilarity judgments remarkably well, often outperforming more complex DNNs. This highlights the power of a statistically derived "face space."
  • Human perception is "statistically tuned": The findings suggest that human face representations are not just sensitive to raw visual differences but are also attuned to the statistical distribution of facial variations in the population, as captured by models like the BFM.
  • Identity tolerance is high: Humans can perceive faces as belonging to the same identity even when they have relatively large Euclidean distances in the BFM, suggesting a high tolerance for changes in appearance.
  • Geometric relationships are less critical than overall distance: While the BFM captures geometric relationships (angles, radii), simple Euclidean distance within this space was a better predictor of human judgments than these specific geometric configurations.
  • DNNs are powerful, but not always superior: State-of-the-art DNNs, particularly VGG-Face, performed very well, but their advantage over BFM distance was not always significant, especially after sigmoidal fitting.

Key Results

  • The BFM Euclidean distance achieved a Pearson correlation of 0.82 with human dissimilarity judgments, and a sigmoidal fit improved this to 0.86.
  • The VGG-Face model (trained on face identification) was the highest-performing model in stimulus set A, achieving a correlation of 0.88 (raw) and 0.91 (sigmoidally fitted).
  • In stimulus set B, VGG-Face still performed well, but was not significantly superior to several other models, including VGG-BFM-identity, VGG-Object, Alexnet, AAM, and GIST.
  • The BFM model was competitive, being outperformed only by VGG-Face and VGG-BFM-identity in stimulus set A, and by VGG-Face in stimulus set B.
  • The combined weighted model did not significantly outperform the best individual models.

Limitations and Future Work

  • Stimulus limitations: The study used faces rendered under controlled lighting and pose conditions, which might favor image-computable models. Future work could explore how these models perform with variations in lighting and viewpoint.
  • BFM limitations: The BFM is based on a limited sample (200 individuals) and may not fully capture the diversity of human faces. The authors suggest that future work could test newer 3D morphable models.
  • Familiarity and experience: The study did not account for the substantial role of familiarity in face recognition. Future research could investigate how familiarity interacts with these models.
  • Dissociating identity and dissimilarity: The current task entangled identity and dissimilarity judgments. Future experiments could aim to dissociate these more clearly.
  • Model dissociation: The authors note that several high-performing models made highly correlated predictions, making it difficult to discriminate between them. They suggest using stimulus optimization methods to identify sets of stimuli that would better differentiate these models.

Summary updated

Abstract

Human vision is attuned to the subtle differences between individual faces. Yet we lack a quantitative way of predicting how similar two face images look, or whether they appear to show the same person. Principal-components-based 3D morphable models are widely used to generate stimuli in face perception research. These models capture the distribution of real human faces in terms of dimensions of physical shape and texture. How well does a “face space” defined to model the distribution of faces as an isotropic Gaussian explain human face perception? We designed a behavioural task to collect dissimilarity and same/different identity judgements for 232 pairs of realistic faces. The stimuli densely sampled geometric relationships in a face space derived from principal components of 3D shape and texture (Basel Face Model, BFM). We then compared a wide range of models in their ability to predict the data, including the BFM from which faces were generated, a 2D morphable model derived from face photographs, and image-computable models of visual perception. Euclidean distance in the BFM explained both similarity and identity judgements surprisingly well. In a comparison against 14 alternative models, we found that BFM distance was competitive with representational distances in state-of-the-art image-computable deep neural networks (DNNs), including a novel DNN trained on BFM identities. Models describing the distribution of facial features across individuals are not only useful tools for stimulus generation. They also capture important information about how faces are perceived, suggesting that human face representations are tuned to the statistical distribution of faces.
BibTeX
@misc{jozwik2021face,
  author = {Kamila M. Jozwik and Jonathan O’Keeffe and Katherine R. Storrs and Wenxuan Guo and Tal Golan and Nikolaus Kriegeskorte},
  title = {{Face dissimilarity judgements are predicted by representational distance in morphable and image-computable models}},
  year = {2021},
  doi = {10.1101/2021.04.09.438859},
  howpublished = {bioRxiv},
}
Figure