TL;DR
This paper investigates how well modern vision transformers, trained without explicit memorability objectives, capture human image memorability. They discover that reconstruction loss from sparse autoencoders trained on these transformer representations is a strong predictor of memorability, outperforming previous methods and generalizing to new datasets.
What Problem Does This Paper Address?
Humans have a remarkable ability to remember images, yet some images are inherently more memorable than others. Understanding what makes an image memorable is a long-standing question in both cognitive science and computer vision. While previous computer vision research has focused on handcrafted features or convolutional neural networks (CNNs), this paper explores whether state-of-the-art vision transformers, particularly those trained for vision-language tasks, implicitly learn to represent image memorability within their internal workings. The goal is to bridge the gap between human perception and AI's understanding of visual memory.
Methods
The researchers explore several internal features of pre-trained vision transformers (CLIP, SigLIP2, and DINOv2) to find correlates of memorability:
- Activations: They extract [CLS] token activations from each layer of the transformers, analyzing mean, maximum, and maximum absolute values. They also compute the "delta" of the [CLS] token's representation between adjacent layers as a proxy for processing depth.
- Attention Entropy: They measure the entropy of the attention distribution applied by the [CLS] token to image patches across different layers, hypothesizing that more focused attention might correlate with memorability.
- Patch Uniformity: They assess how varied image patches are within an image at each layer, based on the idea that less memorable images might lack salient objects.
- Sparse Autoencoders: A key contribution is training sparse autoencoders on the transformer's internal representations. The reconstruction loss from these autoencoders is then used as a proxy for memorability. They experiment with different autoencoder setups, including using representations from CLIP and SigLIP2, and also ViTMAE.
Datasets
- MemCat: This dataset, consisting of 10,000 images across 5 categories (animal, food, vehicle, landscape, sports), was primarily used for training and evaluating the autoencoders. It provides crowdsourced memorability scores.
- LaMem: A larger dataset of 58,741 images was used to test the generalization capabilities of the best-performing autoencoders.
Key Insights and Ideas
- Autoencoder Loss as a Strong Predictor: The most significant finding is that the reconstruction loss from sparse autoencoders trained on transformer representations is a powerful correlate of image memorability, outperforming previous CNN-based approaches. This suggests that images that are "harder to reconstruct" by the autoencoder are more memorable to humans.
- Implicit Memorability in Vision Transformers: Even though these transformers were not explicitly trained to predict memorability, their internal representations contain information that correlates with it.
- Patch Uniformity and Memorability: A negative correlation was observed between patch uniformity and memorability, aligning with cognitive science findings that images with less distinct or salient objects are less memorable.
- Attention Distribution Matters: Attention entropy shows a clearer pattern across layers and models compared to simple activation metrics, suggesting that how the model distributes its attention is informative about memorability.
- Language Supervision May Help: Autoencoders trained on representations from language-guided models (CLIP, SigLIP2) showed stronger correlations than those trained on vision-only models (DINOv2), hinting that language supervision might imbue representations with more memorability-relevant semantics.
- Latent Space Structure: The latent space of the best autoencoder shows that images with high memorability are not necessarily clustered together, but low-memorability images (especially landscapes) tend to form distinct clusters.
Key Results
- Sparse autoencoders trained on CLIP and SigLIP2 representations achieved Spearman's correlation coefficients of up to 0.57 (and even 0.58 with a single-exposure setup) with memorability scores on the MemCat dataset.
- These autoencoders generalized well to the LaMem dataset, achieving correlations of 0.449 (CLIP) and 0.423 (SigLIP2) with aggregated scores, and 0.414 and 0.388 respectively with split scores.
- Linear regression analysis showed that activation mean and attention entropy were strong positive predictors of memorability, while patch uniformity was a negative predictor.
- ViTMAE autoencoders showed very weak correlations, suggesting that pre-trained masked autoencoders might not be directly suitable as off-the-shelf memorability predictors.
Limitations and Future Work
- ViTMAE Limitations: The paper notes that ViTMAE models yielded very weak correlations, suggesting that their training objectives might not align well with capturing human memorability signals.
- Generalization to Other Modalities: The authors suggest future work could explore applying these methods to predict memorability in other modalities, such as language.
- Exploring More Models and Datasets: Future research could investigate a wider range of models, datasets, and incorporate other factors like emotions and context to further understand image memorability.
- Interpreting Latent Dimensions: While they identified features correlating with memorability and visualized images activating them, deeper interpretation of these latent dimensions could be a future direction.
- DINOv3 vs. DINOv2: While DINOv3 showed some promise with register tokens, the paper primarily uses DINOv2 for comparability, leaving the full exploration of DINOv3's impact for future work.
Summary updated