TL;DR
This paper introduces a novel deep learning approach to infer personality traits from social media images, by linking images to personality through associated text tags. They demonstrate that personality can be inferred from images alone, and that this visual inference is consistent with text-based personality analysis.
What Problem Does This Paper Address?
Social media platforms are awash with user-generated content, including billions of images uploaded daily. While text-based analysis of this content has been used to infer user personality, the vast majority of images have remained largely unexploited for this purpose. This paper aims to bridge this gap by exploring whether personality traits can be reliably inferred from images themselves, specifically by leveraging the correlation between images and the text they are shared with. The core question is whether personality is a consistent factor that manifests in both a user's written words and their chosen visual content.
Methods
The authors propose a deep learning methodology to build a combined image-and-text-based personality trait model. Their approach starts by identifying words highly correlated with specific personality traits (based on prior work by Yarkoni et al.). These "personality-related words" are then used as tags to retrieve a large dataset of images from social media. These images, termed "MindPics," are assumed to be associated with the personality trait indicated by their tags.
A Convolutional Neural Network (CNN) is then trained on these MindPics to learn visual features that correlate with personality traits. They explore two main model architectures:
- One model per trait: Five separate CNNs are trained, each dedicated to predicting one of the Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism - OCEAN).
- All-in-one model: A single CNN is used, but with five independent classifiers at the output layer, one for each trait. This approach aims to leverage shared feature extraction layers for efficiency and potentially better generalization.
The models are trained using both random weight initialization and a finetuning approach, where weights are initialized from a model pre-trained on the ImageNet dataset.
Datasets
The primary dataset used in this work is a collection of publicly shared photos crawled from Instagram. This dataset, referred to as MindPics, was curated by using personality-related words as search queries. For each of the five personality traits, approximately 1,100 images were selected per word, resulting in a total of 121,000 images for training. The dataset was split into 80% for training and 20% for testing.
Key Insights and Ideas
- Personality is invariant across text and image domains: The core finding is that personality traits, as reflected in word choice, also manifest in the images users choose to share. This suggests a deeper, underlying personality structure that influences both textual and visual communication.
- Images can be powerful personality indicators: The study demonstrates that by analyzing images associated with specific personality-related tags, it's possible to infer personality traits with significant accuracy, comparable to text-based methods.
- "MindPics" reveal abstract concepts: The qualitative analysis shows that images associated with abstract concepts (e.g., nature, art, emotions) are particularly indicative of certain personality traits, especially for traits like Openness and Neuroticism.
- All-in-one models are more efficient and effective: The "all-in-one" CNN architecture, which shares feature extraction layers across all personality traits, not only reduces computational cost but also achieves slightly better accuracy compared to training separate models for each trait.
- Pre-training on ImageNet significantly boosts performance: Finetuning CNN models pre-trained on the large-scale ImageNet dataset leads to substantial improvements in personality inference accuracy, highlighting the benefit of leveraging general visual features.
Key Results
- Quantitative Accuracy: The best performing model, a finetuned ResNet "all-in-one" model, achieved an average classification accuracy of 71.9% across the five personality traits. The individual trait accuracies ranged from 69.5% (Openness) to 77.7% (Extraversion).
- Comparison to Text-Based Methods: The paper shows that their image-based approach achieves competitive results, with the average accuracy of their best model (71.9%) being comparable to or exceeding some text-only methods reported in the literature (e.g., Golbeck et al. at 61.66%).
- Qualitative Findings:
- High Extraversion was strongly associated with images of people, crowds, concerts, and social settings.
- Low Extraversion was linked to images of cats, books, and knitting.
- High Openness included images of books, the moon, landscapes, and art.
- High Conscientiousness correlated with images of food, especially healthy food, and people exercising.
- High Neuroticism was activated by pets, while Low Neuroticism was associated with landscapes and sunsets.
- t-SNE visualizations showed that images belonging to high and low scores of each trait were generally well-separated, indicating that the model learned discriminative features.
Limitations and Future Work
- Focus on specific traits: The current model infers a specific personality trait conveyed by a single image, rather than the full personality profile of a user.
- Image variability: The study acknowledges the significant intra-class and inter-class variability in images, making it challenging to identify unique patterns for each class.
- Tag reliance: The methodology relies on the correlation between specific words and personality traits, which might not capture all nuances of personality expression.
- Future Work: The authors suggest refining the personality model by considering a wider range of words and images, and importantly, by involving psychology experts to further validate and improve the model. They also propose exploring how to infer the full personality profile of a user by analyzing a collection of their images.
Summary updated