TL;DR
This paper moves beyond measuring how much people differ in their judgments of faces and art to asking what shape those differences take. By using a two-factor generative model, the author demonstrates that while face judgments are mostly a matter of reweighting shared dimensions, aesthetic judgments of art involve a continuous spread of preferences that often cross the population consensus, meaning some people truly value the opposite of the majority.
What Problem Does This Paper Address?
We know that "beauty is in the eye of the beholder," and previous research has quantified this variance—for instance, about half of the variance in face attractiveness ratings is due to personal taste. However, we haven't known the shape of these differences.
Do we all agree on the basic "axes" of judgment (like how "trustworthy" or "abstract" something is) and just differ in how much weight we give them? Or do some of us actually prefer the opposite end of an axis? The author proposes three candidate "shapes" for individual differences:
- Reweighting (D1): Everyone agrees on the direction of an axis, but differs in emphasis.
- Linear Transform/Reversal (D2): Preferences can spread across the zero-point, allowing some people to value the opposite end of a shared axis.
- Stimulus-Specific (D3): Personal quirks that can't be explained by shared dimensions at all.
Methods
The author developed a two-factor generative model that decomposes judgments into stimulus embeddings ($f_i$) and person-specific latent representations ($z_k$).
- Encoder: A permutation-invariant neural network that infers a person's latent "taste" ($z_k$) from just 10–20 of their previous judgments.
- Decoder: Predicts a judgment by combining the stimulus features with the person's latent representation. The decoder has four nested levels (D0–D3) corresponding to the hypotheses above.
- Sign Check: To verify if people actually "reversed" their preferences, the author directly estimated rater weights and compared their signs to the population mean.
This framework allows the model to predict how a new person (not seen in training) would rate a new stimulus, providing a stringent test of the hypotheses.
Datasets
The study utilized ten public datasets encompassing roughly 20,000 raters:
- Faces: Xu et al. (spatial arrangements), London Set (attractiveness), AVFS (similarity triplets), FaceSim3D, and SCUT-FBP5500.
- Art & Photos: LAPIS (aesthetic ratings of 11k+ artworks), AADB, and EVA (photographs).
- Jokes: Jester (funniness ratings).
- Odour: Keller & Vosshall (pleasantness).
- Cross-Domain: XPASS-Vis (same raters judging art, fashion, and landscapes).
Key Insights and Ideas
- Faces vs. Art: Individual differences in face perception are "reweighting-shaped." We generally agree on the underlying dimensions; we just prioritize them differently. In contrast, art preferences are "reversal-shaped."
- The Continuous Tail: In art, people who disagree with the consensus aren't a discrete "rebel" group. Instead, preferences spread continuously along shared axes (like abstract vs. figurative). For about 28% of raters in the LAPIS dataset, this spread crosses zero, meaning they value the opposite of the population mean.
- Taste $\neq$ Demographics: A person's latent representation ($z$) strongly predicts their specific preferences (e.g., a preference for male vs. female faces) but is almost entirely uncorrelated with their self-reported age, gender, or race.
- Efficiency: A person's entire judgment space for complex stimuli like faces can be recovered with surprising accuracy from as few as 20 judgments.
Key Results
- Individualization Gain: Moving from a population-average model (D0) to an individual model (D1) significantly improved prediction accuracy across almost all datasets. For example, in face attractiveness (London Set), Pearson $r$ rose from 0.590 to 0.622.
- Art Reversals: In the LAPIS artwork dataset, the reversal-capable model (D2) significantly outperformed the reweighting model (D1), with a gain of +0.021. The "sign check" confirmed that nearly 10% of raters had a negative cosine similarity with the population weight vector.
- Face Consensus: For faces, the D2 model offered almost no improvement over D1, suggesting that true preference reversals are statistically undetectable in face judgment tasks.
Limitations and Future Work
- Sample Dependency: The size of the "tail" (the proportion of people who reverse the consensus) was found to be a property of the specific rater sample, not just the stimuli. This suggests cultural or demographic factors not captured in the current metadata might play a role.
- Data Breadth: While the face data is extensive, the findings for art and jokes rely on fewer datasets.
- Future Directions: The author suggests that future work should investigate why these shapes differ across domains—perhaps by linking the shape of individual differences to personality traits like the "Big Five," which was not possible with the current datasets. There is also a call to explore the neural basis of these personal weightings.
Summary updated