The shape of individual differences in judgment: shared axes, personal weights, and when their spread crosses the consensus

Hiroshi HigashiarXiv 2026 · arXiv

TL;DR

This paper moves beyond measuring how much people differ in their judgments of faces and art to asking what shape those differences take. By using a two-factor generative model, the author demonstrates that while face judgments are mostly a matter of reweighting shared dimensions, aesthetic judgments of art involve a continuous spread of preferences that often cross the population consensus, meaning some people truly value the opposite of the majority.

What Problem Does This Paper Address?

We know that "beauty is in the eye of the beholder," and previous research has quantified this variance—for instance, about half of the variance in face attractiveness ratings is due to personal taste. However, we haven't known the shape of these differences.

Do we all agree on the basic "axes" of judgment (like how "trustworthy" or "abstract" something is) and just differ in how much weight we give them? Or do some of us actually prefer the opposite end of an axis? The author proposes three candidate "shapes" for individual differences:

  1. Reweighting (D1): Everyone agrees on the direction of an axis, but differs in emphasis.
  2. Linear Transform/Reversal (D2): Preferences can spread across the zero-point, allowing some people to value the opposite end of a shared axis.
  3. Stimulus-Specific (D3): Personal quirks that can't be explained by shared dimensions at all.
The three hypothesized shapes of individual differences and the two-factor model architecture.

Methods

The author developed a two-factor generative model that decomposes judgments into stimulus embeddings ($f_i$) and person-specific latent representations ($z_k$).

  • Encoder: A permutation-invariant neural network that infers a person's latent "taste" ($z_k$) from just 10–20 of their previous judgments.
  • Decoder: Predicts a judgment by combining the stimulus features with the person's latent representation. The decoder has four nested levels (D0–D3) corresponding to the hypotheses above.
  • Sign Check: To verify if people actually "reversed" their preferences, the author directly estimated rater weights and compared their signs to the population mean.

This framework allows the model to predict how a new person (not seen in training) would rate a new stimulus, providing a stringent test of the hypotheses.

Datasets

The study utilized ten public datasets encompassing roughly 20,000 raters:

  • Faces: Xu et al. (spatial arrangements), London Set (attractiveness), AVFS (similarity triplets), FaceSim3D, and SCUT-FBP5500.
  • Art & Photos: LAPIS (aesthetic ratings of 11k+ artworks), AADB, and EVA (photographs).
  • Jokes: Jester (funniness ratings).
  • Odour: Keller & Vosshall (pleasantness).
  • Cross-Domain: XPASS-Vis (same raters judging art, fashion, and landscapes).

Key Insights and Ideas

  • Faces vs. Art: Individual differences in face perception are "reweighting-shaped." We generally agree on the underlying dimensions; we just prioritize them differently. In contrast, art preferences are "reversal-shaped."
  • The Continuous Tail: In art, people who disagree with the consensus aren't a discrete "rebel" group. Instead, preferences spread continuously along shared axes (like abstract vs. figurative). For about 28% of raters in the LAPIS dataset, this spread crosses zero, meaning they value the opposite of the population mean.
  • Taste $\neq$ Demographics: A person's latent representation ($z$) strongly predicts their specific preferences (e.g., a preference for male vs. female faces) but is almost entirely uncorrelated with their self-reported age, gender, or race.
  • Efficiency: A person's entire judgment space for complex stimuli like faces can be recovered with surprising accuracy from as few as 20 judgments.
Distributions of rater weights showing that for art (LAPIS), the distribution crosses zero, indicating reversals, whereas for faces (London Set), it does not.

Key Results

  • Individualization Gain: Moving from a population-average model (D0) to an individual model (D1) significantly improved prediction accuracy across almost all datasets. For example, in face attractiveness (London Set), Pearson $r$ rose from 0.590 to 0.622.
  • Art Reversals: In the LAPIS artwork dataset, the reversal-capable model (D2) significantly outperformed the reweighting model (D1), with a gain of +0.021. The "sign check" confirmed that nearly 10% of raters had a negative cosine similarity with the population weight vector.
  • Face Consensus: For faces, the D2 model offered almost no improvement over D1, suggesting that true preference reversals are statistically undetectable in face judgment tasks.

Limitations and Future Work

  • Sample Dependency: The size of the "tail" (the proportion of people who reverse the consensus) was found to be a property of the specific rater sample, not just the stimuli. This suggests cultural or demographic factors not captured in the current metadata might play a role.
  • Data Breadth: While the face data is extensive, the findings for art and jokes rely on fewer datasets.
  • Future Directions: The author suggests that future work should investigate why these shapes differ across domains—perhaps by linking the shape of individual differences to personality traits like the "Big Five," which was not possible with the current datasets. There is also a call to explore the neural basis of these personal weightings.

Summary updated

Abstract

People differ in how they judge faces, artworks, photographs and jokes. Previous work measured the size of these differences; we ask what shape they take. We wrote three candidate shapes as nested generative models (same-sign reweighting of shared dimensions, a linear transform that allows reversal, and stimulus-specific offsets) and compared how well they predict people unseen in training, using ten public datasets (about 20,000 raters). The models recovered a person's judgment space from 10-20 judgments and predicted judgments of unseen stimuli. For faces, reweighting was sufficient, and weights opposing the population were not statistically detected. For artworks, weights spread continuously along the shared abstract-figurative direction, and 28 percent of raters fell on the opposite side from the population. This tail was much smaller when a different rater sample judged the same artworks. Individual differences thus differ in shape as well as size, and the shape depends on domain and raters.
BibTeX
@misc{higashi2026shape,
  author = {Hiroshi Higashi},
  title = {{The shape of individual differences in judgment: shared axes, personal weights, and when their spread crosses the consensus}},
  year = {2026},
  eprint = {2611.22440},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22440},
}
Figure