Large-Scale Mapping of Human Activity using Geo-Tagged Videos

Yi Zhu, Sen Liu, Shawn Newsam

Abstract: This paper is the first work to perform spatio-temporal mapping of human activity using the visual content of geo-tagged videos. We utilize a recent deep-learning based video analysis framework, termed hidden two-stream networks, to recognize a range of activities in YouTube videos. This framework is efficient and can run in real time or faster which is important for recognizing events as they occur in streaming video or for reducing latency in analyzing already captured video. This is, in turn, important for using video in smart-city applications. We perform a series of experiments to show our approach is able to accurately map activities both spatially and temporally. We also demonstrate the advantages of using the visual content over the tags/titles.

Introduction

Understanding how people move and interact within a city is vital for "smart city" applications like traffic management, disease control, and resource allocation. While researchers have previously used GPS trajectories or mobile phone records to map mobility, these data sources lack semantic context—they don't tell us what people are actually doing. This paper bridges the gap between high-level computer vision and geographic information science by leveraging geo-tagged videos as a rich medium for human activity mapping.

Methodology: Hidden Two-Stream Networks

The core technical contribution is the use of a Hidden Two-Stream Network for activity recognition. Standard "two-stream" networks are often bottlenecked by the need to pre-compute optical flow for motion analysis. The "Hidden" approach bypasses this by integrating MotionNet, an unsupervised CNN that estimates optical flow between consecutive frames in real-time.

The Hidden Two-Stream Network architecture, featuring spatial and temporal streams with an integrated MotionNet for real-time optical flow estimation.

As shown in the architecture diagram, the system consists of a Spatial Stream (analyzing RGB frames for objects/scenes) and a Temporal Stream (analyzing motion). The outputs are combined via Late Fusion. This architecture allows the system to run at 130.56 fps, more than four times faster than standard real-time video, making it feasible to process hundreds of thousands of videos at scale.

Large-Scale Mapping in San Francisco

The framework was validated using a massive "in-the-wild" dataset of 265,477 geo-tagged YouTube videos uploaded within San Francisco in 2016. The system achieved 90.94% accuracy on a 10-class validation set, outperforming popular models like C3D.

Spatio-temporal mapping of six sports activities across San Francisco, demonstrating the framework's ability to localize specific human activities using video content.

Key Insights and Geographic Discovery

  • Visual Context vs. Metadata: Visual content proved far more reliable than video tags. For example, the classifier correctly identified "soccer" videos that were linguistically mislabeled as "football" due to regional naming conventions.
  • Indoor Discovery: The system successfully mapped a golf activity to a building in downtown San Francisco. While satellite imagery showed only a roof, the ground-level video revealed an indoor driving range, highlighting video's ability to perform discovery where aerial sensors are "blind."
  • Geospatial Validation: The system reconstructed the route of the SF Pride Parade and found that detections of "street fights" correlated with official police records for assaults in high-crime districts.

Conclusion and Future Work

This work demonstrates that geo-tagged videos are a powerful source for urban sensing. While successful, the authors note that generalizing handheld "YouTube-like" training data to fixed-angle surveillance cameras remains a challenge. Future work involves compressing these CNN models for edge deployment directly on mobile devices or smart networked cameras.

Summary updated

Abstract

This paper is the first work to perform spatio-temporal mapping of human activity using the visual content of geo-tagged videos. We utilize a recent deep-learning based video analysis framework, termed hidden two-stream networks, to recognize a range of activities in YouTube videos. This framework is efficient and can run in real time or faster which is important for recognizing events as they occur in streaming video or for reducing latency in analyzing already captured video. This is, in turn, important for using video in smart-city applications. We perform a series of experiments to show our approach is able to accurately map activities both spatially and temporally. We also demonstrate the advantages of using the visual content over the tags/titles.
BibTeX
@misc{zhu2017large,
  author = {Yi Zhu and Sen Liu and Shawn Newsam},
  title = {{Large-Scale Mapping of Human Activity using Geo-Tagged Videos}},
  year = {2017},
  eprint = {1706.07911},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.07911},
}
Figure