Understanding and Predicting the Memorability of Outdoor Natural Scenes

Jiaxin Lu, Mai Xu, Ren Yang, Zulin WangIEEE TIP 2020 · OpenReview

Abstract

Memorability measures how easily an image is retained in memory after a brief glance. While previous research has identified visual features that make objects or faces memorable, predicting the memorability of outdoor natural scenes remains a challenge due to the lack of distinct focal points. This paper introduces the Large-scale Natural Scene Image Memorability (LNSIM) database and DeepNSM, an end-to-end deep neural network that leverages scene category information to significantly improve memorability prediction for natural landscapes.

The Challenge of Natural Scene Memorability

Most image memorability research focuses on "object-centric" images where specific subjects—like people or animals—act as memory anchors. Outdoor natural scenes, such as deserts or mountain ranges, lack these salient anchors, making them harder for traditional models to evaluate. Existing state-of-the-art models like MemNet often struggle with these "background-only" images because they cannot easily identify which regions are worth remembering.

The LNSIM Dataset

To address this gap, the authors established the LNSIM database, the largest of its kind. It contains 2,632 images across 71 categories. Ground-truth memorability scores were collected using a "memory game" protocol where at least 80 participants per image identified repeated visuals in a stream of images.

A spectrum of outdoor natural scene images ranked by their ground-truth memorability scores in the LNSIM dataset, ranging from highly memorable (0.95) to less memorable (0.14).

DeepNSM: A Category-Aware Architecture

The core innovation of this research is the DeepNSM (Deep Natural Scene Memorability) predictor. Through feature analysis, the authors discovered that high-level scene categories are the strongest predictors of memorability in natural landscapes. DeepNSM utilizes a dual-branch architecture:

  1. Baseline Branch: A fine-tuned MemNet that extracts general spatial and visual features.
  2. Category Branch: A ResNet-based module fine-tuned on the "Places" dataset to identify specific scene types (e.g., glacier, marsh).

By fusing these generic deep features with category-specific data, the model can better understand the "macroscopic" context that the human brain uses to encode memories of environments.

Key Insights: The Role of Scene Category

A striking finding of the study is the correlation between memorability and rarity. Scenes that are rarely encountered in daily life or described less frequently in literature—such as an aurora or a unique rock arch—tend to be significantly more memorable than common landscapes like forests or mountains.

Sample images from the LNSIM dataset grouped by category, showing how specific scene types (e.g., Aurora) consistently rank higher in memorability than others (e.g., Snow Mountain).

Furthermore, the research suggests that standard visual saliency (what catches the eye) does not equate to memorability in natural scenes. A visually striking element might attract immediate attention without making the overall scene stick in the viewer's mind.

Performance and Results

DeepNSM significantly outperforms previous methods. On the LNSIM dataset, it achieved a Spearman Rank Correlation Coefficient (SRCC) of 0.58, compared to 0.43 for MemNet and 0.15–0.33 for traditional handcrafted features. The model also demonstrated strong generalization, maintaining an SRCC of 0.55 when tested on independent datasets. With human consistency on this task measured at 0.78, DeepNSM represents a major step toward human-level performance in predicting subjective image memorability.

Conclusion

By shifting the focus from object-level saliency to macroscopic scene categories, this research provides a new framework for understanding how we remember the natural world. While challenges remain—such as handling low-contrast images or repeating patterns—the LNSIM dataset and DeepNSM model provide essential tools for fields ranging from tourism marketing to automated image curation.

Summary updated

Abstract

Memorability measures how easily an image is to be memorized after glancing, which may contribute to designing magazine covers, tourism publicity materials, and so forth. Recent works have shed light on the visual features that make generic images, object images or face photographs memorable. However, these methods are not able to effectively predict the memorability of outdoor natural scene images. To overcome this shortcoming of previous works, in this paper, we provide an attempt to answer: “what exactly makes outdoor natural scenes memorable”. To this end, we first establish a large-scale outdoor natural scene image memorability (LNSIM) database, containing 2,632 outdoor natural scene images with their ground truth memorability scores and the multi-label scene category annotations. Then, similar to previous works, we mine our database to investigate how low-, middle- and high-level handcrafted features affect the memorability of outdoor natural scenes. In particular, we find that the high-level feature of scene category is rather correlated with outdoor natural scene memorability, and the deep features learnt by deep neural network (DNN) are also effective in predicting the memorability scores. Moreover, combining the deep features with the category feature can further boost the performance of memorability prediction. Therefore, we propose an end-to-end DNN based outdoor natural scene memorability (DeepNSM) predictor, which takes advantage of the learned category-related features. Then, the experimental results validate the effectiveness of our DeepNSM model, exceeding the state-of-the-art methods. Finally, we try to understand the reason of the good performance for our DeepNSM model, and also study the cases that our DeepNSM model succeeds or fails to accurately predict the memorability of outdoor natural scenes.
BibTeX
@article{lu2020understanding,
  author = {Jiaxin Lu and Mai Xu and Ren Yang and Zulin Wang},
  title = {{Understanding and Predicting the Memorability of Outdoor Natural Scenes}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=5bg6AqO56Wj},
}

Loading related content...

Figure