TL;DR
This paper demonstrates that purely perceptual features learned by deep neural networks can predict a significant majority of human affective responses (arousal, valence, beauty) to images, suggesting that "seeing" plays a more fundamental role in "feeling" than previously thought. By analyzing a diverse set of "affectless" machine vision models, the study reveals that the richness of learned visual representations, rather than explicit affective training, is the key driver of this predictive power.
What Problem Does This Paper Address?
The fundamental relationship between perception and affect—how we "see" and "feel" the world—is a long-standing question in psychology and neuroscience. Traditional theories often emphasize the interplay of perception, physiological responses (like heartbeat), and cognitive interpretation in generating feelings. However, it's been challenging to disentangle the precise contribution of pure perception to affect, especially given the integrated nature of human experience. This paper aims to isolate the perceptual component by leveraging machine vision models, which can "see" without the biological or cognitive machinery that typically underpins human affect. The goal is to quantify how much of our visually evoked affect can be explained by the way machines represent visual information, and to understand what aspects of this representation learning are most crucial.
Methods
The core methodology involves using a large survey of 180 state-of-the-art deep neural network (DNN) models. These models, trained on standard computer vision tasks (like object recognition) and devoid of any affective training or physiological simulation, serve as "affectless" perceptual systems.
- Feature Extraction: For each model, features were extracted from every computational layer.
- Linear Decoding: These extracted features were then used as input to linear regression models. The output of these decoders was trained to predict human-provided ratings of arousal, valence, and beauty for a given image. This process was performed without further learning on the DNN features themselves, essentially treating them as fixed inputs.
- Model Zoology: A diverse set of models was used, varying in architecture, training objective (category-supervised, self-supervised, untrained), and depth of layers. This "model zoology" approach allows for a systematic investigation of which types of perceptual representations are most predictive of affect.
- Cross-Validation and Scoring: Leave-one-out cross-validation was used for regression fitting. Predictive accuracy was measured using Pearson correlation ($r$) and "explainable variance explained" ($r_{EVE}$), which normalizes accuracy by human interrater reliability (noise ceilings).
Datasets
- OASIS (Open Affective Standardized Image Set): A dataset of 900 images across four categories (people, animals, objects, scenes) with arousal and valence ratings from 822 human respondents, and beauty ratings from 751 respondents.
- Vessel Dataset: A dataset of 562 images across five categories (art, faces, landscapes, internal architecture, external architecture) with beauty ratings.
The paper does not introduce new benchmark datasets but utilizes existing ones to evaluate the predictive power of machine vision models on human affective judgments.
Key Insights and Ideas
- Perception as a Foundation for Affect: The most striking finding is that purely perceptual models, trained solely on visual data without any affective component, can explain a substantial majority of the variance in human affective responses. This suggests that our feelings about what we see are deeply rooted in how our brains (or machines) represent visual information.
- Representation Learning is Key, Not Just Raw Statistics: The predictive power isn't solely due to models efficiently encoding low-level image statistics. Instead, it's the learned representations—the hierarchical abstractions built from diverse visual experiences—that drive affect prediction. This highlights the importance of how visual information is structured and organized.
- Deep Layers Matter Most: Affective prediction accuracy generally increases with the depth of the neural network layer from which features are extracted. This implies that more abstract, higher-level visual representations are more strongly linked to affective experience.
- Training is Crucial: Randomly initialized (untrained) models show very low predictive accuracy, underscoring that the learning process itself, not just the architecture, is essential for developing affect-predictive representations.
- Self-Supervision vs. Category Supervision: Models trained with self-supervised learning (learning from the structure of the data itself) are as effective at predicting affect as those trained with explicit category labels. This suggests that explicit "top-down" semantic learning isn't necessary for generating affect-predictive features.
- Beauty is Highly Predictable: Affective ratings for "beauty" are consistently more predictable by these models than arousal or valence, suggesting a particularly strong link between perceptual representation and aesthetic judgment.
Key Results
- Majority of Variance Explained: Purely perceptual models explain a majority of the explainable variance in human affective ratings. The most predictive models achieve an $r_{EVE}$ of up to 0.729 (over 70% of explainable variance).
- Average Model Performance: On average, object recognition models predict group-level affect ratings significantly above human "shared taste" ceilings and over halfway to the overall "noise ceiling."
- Category-Specific Performance: Landscapes and faces are more predictable categories for affect prediction than social scenes or art.
- Individual vs. Group Prediction: While models predict group averages well, their ability to predict individual human ratings varies, with "taste-typicality" (how closely an individual's ratings align with the group average) being a strong predictor of model predictability.
- Cross-Dataset Generalization: Models trained on one dataset can predict affect in another, particularly for similar image categories like landscapes, demonstrating some transferability of learned representations.
- Hybrid Models (CLIP): Vision-language models like CLIP show improved prediction over purely visual models, suggesting language might play a role, though the authors caution this could also be due to larger training datasets.
Limitations and Future Work
- "Affectless" Models: While these models lack explicit affective training, the authors acknowledge that the data they are trained on (e.g., ImageNet) might implicitly contain affective information. Hybrid vision-language models (like CLIP) further blur this line due to their training on text-image pairs, which can include affective language.
- Individual Taste Variation: The models' ability to predict individual human affect ratings is less consistent than for group averages, highlighting the idiosyncratic nature of personal taste and the difficulty in capturing it fully. The authors note that the "taste-typicality" of a respondent is a key factor.
- Future Work: The authors suggest further investigation into the role of language in affect prediction, exploring more controlled comparisons between vision-language models and purely visual models trained on larger, more diverse datasets. They also propose deeper analysis into the specific features and representational structures within deep networks that drive affective predictions, potentially by examining the "surprise" or prediction error signals within these models. Further work is also needed to fully demystify the precise computational mechanisms linking perception to affect.
Summary updated