Social media and deep learning capture the aesthetic quality of the landscape

Ilan Havinga, Diego Marcos, P.w. Bogaart, Lars Hein, Devis TuiaScientific 2021 · DOI

Abstract

Peoples' recreation and well-being are closely related to their aesthetic enjoyment of the landscape. Ecosystem service (ES) assessments record the aesthetic contributions of landscapes to peoples' well-being in support of sustainable policy goals. However, the survey methods available to measure these contributions restrict modelling at large scales. As a result, most studies rely on environmental indicator models but these do not incorporate peoples' actual use of the landscape. Now, social media has emerged as a rich new source of information to understand human-nature interactions while advances in deep learning have enabled large-scale analysis of the imagery uploaded to these platforms. In this study, we test the accuracy of Flickr and deep learning-based models of landscape quality using a crowdsourced survey in Great Britain. We find that this novel modelling approach generates a strong and comparable level of accuracy versus an indicator model and, in combination, captures additional aesthetic information. At the same time, social media provides a direct measure of individuals' aesthetic enjoyment, a point of view inaccessible to indicator models, as well as a greater independence of the scale of measurement and insights into how peoples' appreciation of the landscape changes over time. Our results show how social media and deep learning can support significant advances in modelling the aesthetic contributions of ecosystems for ES assessments.

Introduction

Measuring the aesthetic quality of a landscape is a cornerstone of Ecosystem Service (ES) assessments, which help policymakers understand how nature contributes to human well-being. Traditionally, this is achieved through expensive, small-scale social surveys or "indicator models" that use environmental proxies, such as land cover types, to estimate quality. However, indicator models are often "blind" to actual human interaction and rely on static datasets that fail to capture ephemeral beauty or specific local contexts. This research explores whether the massive, unstructured data available on social media can fill this gap by providing a direct measure of "revealed preferences"—what people actually choose to photograph and share.

Methodology

The researchers developed a multi-stage pipeline to compare and combine social media data with traditional environmental indicators across Great Britain.

Figure 1: (a-e) Deep learning pipeline for predicting scenicness from Flickr images; (f) Comparison of scenicness maps generated by Flickr data versus traditional environmental indicators across Great Britain.

Computer Vision Stack

The study processed 9.8 million geo-located Flickr images using two deep learning architectures based on ResNet-50. The first identified scene classes (e.g., "tundra") and attributes (e.g., "natural light"), while the second was fine-tuned on the "Scenic-Or-Not" (SoN) database—a crowdsourced dataset of 1.5 million ratings—to predict scenicness scores (1–10). This process is illustrated in Figure 1a-e.

Environmental Indicator Stack

A parallel model calculated 41 variables per grid cell, including ecosystem types from European Environment Agency maps, relief/elevation from Digital Elevation Models, and historical points of interest from OpenStreetMap.

Model Fusion

Using Random Forests, the team predicted the average scenicness of 5 km grid cells. They tested three versions: Flickr-only, Indicator-only, and a Combined model. They also employed temporal analysis to track how specific attributes, such as snow cover, influenced scenicness scores across different seasons.

Key Results and Insights

The study found that deep learning models applied to social media data provide a more continuous and nuanced "human-eye" view of the landscape compared to the sharp boundaries often produced by indicator models tied to land-cover polygons.

  • Comparative Accuracy: The Flickr-based model achieved a Kendall’s $\tau$ of 0.683, performing nearly as well as the traditional indicator model (0.730). The Combined model outperformed both ($\tau = 0.739$), suggesting that social media data is most powerful when anchored by physical environmental data.
  • Capturing Local Context: The Flickr model was significantly better at identifying specific high-value spots, such as Achmelvich Bay, and negative contexts, such as industrial areas near Heathrow Airport, which coarse 25m-resolution land-cover maps often overlook (see arrows in Figure 1f).
  • Temporal Dynamics: One of the most striking findings was the model's ability to track "ephemera." There was a high correlation (Pearson’s $R = 0.77$) between the prevalence of "snow" in Flickr photos and actual remotely-sensed snow cover data, allowing for dynamic ecosystem accounting that traditional models miss.

Discussion and Limitations

While social media offers a rich, scale-independent data source, it is not without bias. Flickr users tend to skew toward specific demographics (e.g., 40–60-year-old males), and the models were trained on a specifically British representation of beauty. Future work could explore other platforms like Instagram or TikTok to broaden demographic perspectives and test the model's applicability to different biomes, such as tropical or arid regions. Despite these limitations, the integration of social media and deep learning represents a significant advance in capturing the aesthetic contributions of ecosystems for large-scale assessments.

Summary updated

Abstract

Peoples' recreation and well-being are closely related to their aesthetic enjoyment of the landscape. Ecosystem service (ES) assessments record the aesthetic contributions of landscapes to peoples' well-being in support of sustainable policy goals. However, the survey methods available to measure these contributions restrict modelling at large scales. As a result, most studies rely on environmental indicator models but these do not incorporate peoples' actual use of the landscape. Now, social media has emerged as a rich new source of information to understand human-nature interactions while advances in deep learning have enabled large-scale analysis of the imagery uploaded to these platforms. In this study, we test the accuracy of Flickr and deep learning-based models of landscape quality using a crowdsourced survey in Great Britain. We find that this novel modelling approach generates a strong and comparable level of accuracy versus an indicator model and, in combination, captures additional aesthetic information. At the same time, social media provides a direct measure of individuals' aesthetic enjoyment, a point of view inaccessible to indicator models, as well as a greater independence of the scale of measurement and insights into how peoples' appreciation of the landscape changes over time. Our results show how social media and deep learning can support significant advances in modelling the aesthetic contributions of ecosystems for ES assessments.
BibTeX
@inproceedings{havinga2021social,
  author = {Ilan Havinga and Diego Marcos and P.W. Bogaart and Lars Hein and Devis Tuia},
  title = {{Social media and deep learning capture the aesthetic quality of the landscape}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-99282-0},
}
Figure