Deep Neural Networks and Kernel Density Estimation for Detecting Human Activity Patterns from Geo-Tagged Images: A Case Study of Birdwatching on Flickr

Wei Shao, Caglar Koylu, Chang Zhao, Wei Shao 0008Isprs Int. J. geo inf 2019 · DOI

TL;DR

This paper introduces a novel framework combining deep learning (YOLOv3) for object detection with kernel density estimation to map human activity patterns from geo-tagged social media images. By analyzing millions of Flickr photos of birds, they demonstrate this approach can more accurately identify and map birdwatching activities than traditional keyword searches, offering insights into user behavior and data biases.

What Problem Does This Paper Address?

Extracting meaningful human activity patterns from the vast amount of geo-tagged social media data is a significant challenge. Traditional methods often rely on image metadata (tags, descriptions), which can be incomplete, inaccurate, or misleading. While computer vision, particularly deep learning, offers a way to analyze image content directly, there's a need to evaluate its effectiveness for large-scale human activity analysis and to understand the inherent biases in social media data compared to more structured datasets. This paper specifically addresses this by using birdwatching as a case study, aiming to quantify the utility of deep learning for inferring activities and to compare the patterns derived from social media with those from a dedicated citizen science platform.

Methods

The core of the proposed framework involves two main components:

  • Object Detection with YOLOv3: The authors employ YOLOv3 (You Only Look Once), a state-of-the-art real-time object detection system. YOLOv3 uses convolutional neural networks (CNNs) to directly predict bounding boxes and class probabilities for objects within an image, analyzing visual features like color, shape, and texture. This approach bypasses the limitations of relying solely on metadata.
  • Kernel Density Estimation (KDE): To analyze the spatial and temporal patterns of detected activities, the researchers use KDE. They apply both fixed-distance and adaptive bandwidth KDE.
  • Fixed-distance KDE was used to compare the spatial distributions of images identified by YOLO and metadata search, and by YOLO and eBird data. A bandwidth of 20 miles was chosen.
  • Adaptive KDE was employed to account for user contribution bias and the "small area problem" (unreliable estimates in low-density areas). This method adjusts the bandwidth based on the number of distinct users within a neighborhood, using a threshold of 100 Flickr and eBird users.

The framework involves:

  1. Collecting geo-tagged Flickr images and eBird observations within the conterminous US.
  2. Using YOLOv3 to detect "bird" objects in Flickr images.
  3. Comparing YOLO's bird detection results with metadata keyword searches for verification.
  4. Comparing the spatial and temporal patterns derived from YOLO-detected Flickr images with eBird observations for validation.

Datasets

  • Flickr: Approximately 20 million geo-tagged images collected over a three-year period (December 2013 to December 2016) within the conterminous US. This dataset includes image metadata (ID, user info, location, time, tags, description, title) and the images themselves.
  • eBird: A freely available, global citizen science online bird observation dataset. The study utilized eBird data for the same three-year period and geographic region, containing information on species observed, counts, location, time, and observer details.

The paper does not introduce a new benchmark dataset but rather uses existing large-scale datasets to evaluate its proposed framework.

Key Insights and Ideas

  • YOLOv3 Outperforms Metadata Search: Deep learning object detection (YOLOv3) is significantly more effective than traditional metadata keyword searches for identifying specific objects like birds in images. This is crucial for capturing activities where users might not use precise keywords.
  • Social Media vs. Citizen Science Behavior: The study highlights distinct patterns between Flickr users (casual birdwatchers, often tourists) and eBird users (serious birdwatchers). Flickr patterns are more spatially diverse and less concentrated, suggesting a broader engagement with nature, while eBird patterns are more focused, indicating dedicated observation in specific locations.
  • Bias in Data Sources: Both social media and citizen science data have inherent biases. Flickr's bias leans towards urban areas and general tourism, while eBird's bias can be towards specific species or protected areas due to user motivations. Understanding these biases is key to interpreting activity patterns.
  • Spatial Patterns Reveal Habitat Preferences: Comparing the spatial distributions of Flickr and eBird data, even with their differing magnitudes, reveals similar preferences for coastal areas, populous regions, and natural landscapes, suggesting common areas of interest for bird-related activities.

Key Results

  • Increased Detection: YOLO detected 747,015 (3.8%) images with birds, while metadata search identified 534,121 (2.7%). YOLO increased bird detection by over 50% compared to metadata search.
  • Accuracy of YOLO: Human classification of images detected only by YOLO yielded an accuracy of 89%, compared to only 26% for images detected only by metadata search.
  • Correlation: Spearman Rank correlation showed a strong overlap between eBird observations and YOLO-detected Flickr images, with coefficients of 79% for observation counts and 85% for distinct users.
  • Temporal Trends: Flickr showed a declining trend in bird photographs and users from 2013-2016, consistent with overall platform usage decline. eBird, conversely, showed an increasing trend. Both datasets peaked in spring.
  • Spatial Overlap: Fixed-distance density maps showed substantial overlap between YOLO-detected Flickr images and eBird observations, with both datasets highlighting activity around coastal areas and populous regions. YOLO identified more bird images in urban areas and showed more continuous habitat regions.

Limitations and Future Work

  • Limitations:
    • Inaccurate Classifications: YOLO still produced some misclassifications (e.g., butterflies, flowers, drawings classified as birds), though generally with lower confidence scores.
  • Metadata Bias: The study acknowledges that even with YOLO, the underlying Flickr data is subject to user-generated biases (demographics, behavior, location choices).
  • Small Area Problem: While adaptive KDE was used, the "small area problem" can still affect estimates in very low-density observation areas.
  • Future Work:
    • Comprehensive Accuracy Evaluation: Complete accuracy evaluation of all images classified by both metadata and YOLO.
  • Explore Other Object Detection Libraries: Compare YOLO's performance with other deep learning object detection algorithms.
  • Broader Activity Analysis: Extend the framework to identify characteristics of locations based on detected objects over time, examining place semantics and identifying similarities across different locations globally.
  • Mobility Pattern Analysis: Further investigate the mobility patterns of eBird and Flickr users to understand the dynamics of birdwatching activities more deeply.

Summary updated

BibTeX
@article{shao2019deep,
  author = {Wei Shao and Caglar Koylu and Chang Zhao and Wei Shao},
  title = {{Deep Neural Networks and Kernel Density Estimation for Detecting Human Activity Patterns from Geo-Tagged Images: A Case Study of Birdwatching on Flickr}},
  journal = {Isprs Int. J. geo inf},
  year = {2019},
  doi = {10.3390/ijgi8010045},
}
Figure